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> Mary Ann Millsap 

National Institute of Education 

Over the past decade, increased emphasis and rehance have ^ 
been placed on the educationarroles of testing, assessment, and 
evaluation. Issues raised in these domains reflect a broad range of 
concerns, from methodological concerns about the validit> and 
reliabilit> of measures to political concerns about the appropriate 
and equitable use of technical information, from assignment xifVstu- 
dents in classrooms to federal polic> in the appropriations ofpil- 
lions of dollars for educational programs. They thus have relevance 
for an equall> broad range of decision-making levels — local, state, 
and federal. 

For the past five years, the National Institute of Education 
(NIE), the research and development arm of the U.S. Department 
of Education, has funded a number of studies on these issues. 
Included among its research activities are studies on how testing 
and evaluation information is used in the schools, on the impact 
of testing programs, on the relationship of testing and evaluation 
to instruction, and on new methods for assessing educational out- 
comes. Workshops and conferences on testing and evaluation for 
educators and the general public have been conducted, resulting 
in publications targeted for la> audiences as well as for the research 
community. 

The research agenda arose in response to the unprecedented 
growth in testing and evaluation in schools over the past 15 years 
and because of the absence of systematic information about the 
nature and impact of those activities. The Center for the Study of 
Evaluation at UCLA recognized eacly the need to explore how 
school districts use testing and evaluation information. With NIE 
funding, CSE undertook in 1977 the first national survey of eval- 
uation and testing offices in school districts. This work was pri- 
,marily descriptive in nature, focusing on how such offices were 
organized, what their budgets were, what activities they under- 
took, and whom^they identified as their major clients.. Working in 
conjunction with faculty from the University of California's Los 
Angeles and Berkele> campuses, CSE incorporated into the survey 
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sevenJ item^ relating to school districts as social organ izaTioiTSs^nd 
the roles of stafiF within such organizations. These items, buttressed 
with the considerable sociological literature on organizations, form ^ 
the basis of the papers contained within this monograph. 

The monograph makes an important contribution to the field of 
testi-ng and evaluation use. The papers provide considerable insight, 
from a theoi:etical, sociological perspective, into the incentives and 
dismcentives for using testing and evaluation infonnationjn schools 
and into the role conflicts faced by the providers of such infor- 
mation. From a careful reading of these papers, .the editors extrap- 
olate the changes and modifications that are needed so that technical 
information can be more effectively used within school and school 
district contexts. 

The monograph also complements other NIE supported research 
in testing and evaluation. . Both the Center for the Stud>^ of Eval- 
uation and the Huron Institute are conducting intensive case 
studies in school districts around the country', particularly in those 
school districts identified as successful!^' t>ing evaluation and testing 
information to administrative decisiStimaking and instructional 
reform. The rich data bases from these studies are providing much 
needed information about the Issues faced by a variet>' of educa- 
tors— simerintendents, school boards, principals, program man- 
agers, and teachers — and about how -testing and evaluation 
information has Assisted in addressing these issues. The findings 
from this re^>earch are, being -translated into technical assistance 
strategies to assist other school districts in *the better use of eval- 
uation and testing information. 

In addition to funding these studies, NIE also supports an 
annual grants program' in testing and evaluation, designed pri- 
marily for school districts to conduct their own research on testing 
and evaluation issues within tbeir school districts. ^ 

6y funding interdisciplinary work such as this monograph on the 
orgas^zational factors influencing evaluation and test use, by 
increasing field based research on how district's use evaluation and 
testing information, and b> encouraging school districts to submit 
proposals* of their own design into^j^^IE-funded grants announce- 
ments, NIE is attempting to increase the utility of testing and eval- 
uation inforjnatiOn for more effective schools. 
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This monograph has grown out of. the Center^for, the Study of 
Evaluation's (CSE) on-goirig interest in tlie conduct of educational 
evaJjLiatio^^ how it opferates \t\ school districts, and how it contrib- 
utes to educational*practice. The \\\\^—Etaluai\on in School Dis- 
tricts. Organizatiojial Perspectives — we believe, conveys the 
orientation of the rnonograph and its content. ' y 
j^'^^he monograph .focuses on evaluation and how it occurs within 

^ a specific settihg. school districts' central administrative offices. 
, --This narrows our interest from evaluation-in-general to evaluation- 
" in-operation wrthin this special context. Thu^^ this volume exam- 
ines the relationships between evaluation activities and school dis- 
trict orgcmizational features that impinge upon such activfties. 
Research on evaluation methodolog> and evaluation use often pays 
lip service to the need to understand the context within which 

? evaluation occurs. But th^re are ver> few research ^studies which 
attempt tb relate organizational variables such as decision making, 
internal administrative structures, role definitions, and the nature 
of school districts' "technical core" to the way in which district 
personnel conceive, conduct, and utilize educational evaluation. 
Throughout, the monograph offers nlultiple pferspectives on dis- 
- trict organizations rather than advocating a single. perspective. The 
chapters repr^ent a range of sociological views on evaluation mid 
its function in relation* to other school district operations. Each of 
the chapter authors worked wrfh CSE stafF^members (Jikconcep- 
tualizing a national survey of/^h^ool district evaluation, units and 
in analyzing the resultant data. Findings trom the sur^y itself, 
•which describes evaluation work as it is carried out in acnobl dis- 
tricts, are summ?iri2ied later in this introduction. / 
7 While thef chapters reflect this common dat^l^as^^ they differ 
in approach. Each of the authors agreed either tcyuse the CSE 
survey data as a stimulus for inflecting on evaluatjeiji utilization or 
performed additional analyses of the data to examine their own 
hypotheses about evaluation utilization and the functioning of 
educational evaluation unit6.The autliors are Chades O'Reilly III, 
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UC Berkeley Graduate ScKool of Business AdministraUon; Lynne, 

Zucker UCLA Department. of Sociology; Oscar G'rusky^ UCLA 
Department of Sociology;' and David O'Shea,. UCLA GraduateT 
- School of Education. . " ■ 

. ©•Reilly's comprehensive review of the decision making research - 
literature reminds evaluators of the preeminence of the human 
factor 'in decision making. School district decision makers ike 
those in other organizations, ^e not ratiqnal, consistent, and log- 
ical- they are not disposed to make decisions on tlVe basis ot hard 
data. Instead thev usually consider many factors, oilly one of which 
may. be eyaluatiV^ data, and they X)ften arrive at decisions that 
reflect organizational predilections for their own selt-interests. 
Given the decision makers' organizational constraints, they com- ^ 
monV aninJinierstandably use evaluations, or selectfed portions, 
to justify \vh'at they already had planned'to do, toi interpret data 
to be supportive of aohffeved outcomes. Sometimes they even 
refute pr ignore data that-rtm counter to their preferences. 
, Zucker, through her reanalysfs of the CSE survey data, illus- 
trates how school districts utilize evaluations to perform a ^g- 

■ naling" function to the external environitient. .This signaling 
behavior, typical of organizations which must respond to societal 
demands, leads \o evaluations that are primarily reactive to the 
information needs of external funding agencies. ''Because ol the 
potential or actual conflict among these external dvnensions, she 
assert, most evaluations appear ambiguous and offer'only a narrow 
range of information; the usejlilness. of such evaluations tor ^ 
informing districts' internal decisions about educational pfograms 

.» is thereby diminished. .-^u 
Grusky combines his analysis of CSE s survey results with the 
literature on role conflict and roje ambiguity to point out charac- 
teristics of complex organizations ■ that evaluation unit directors 

- must consider if their work is to influence organisational decision. 

• making. His analysis contains organizatiohal, structural, and 
behavioral iipplications for individuals who fead, school district 
evaluation units and for others who may be involved in designing 
and conducting local evaluation*. " ' r 

O'Shea, finally, discusses some of the a*»*^ced6nts ot current 
' evaluation practice and,places ,the CSE .purvey findings in a his- 
torical context. fle.reminds us of the somewhat arbitrary manner 
in which legislative requirement's for evaluation took torpi. U bhea . 
further develops a theme explicit or implicit in each ot the pre- 

• ceding chapters: that school districts are institutional rather than 
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technical organizations and thus their evaluation imperatives are 
externally focused. This point, when coupled with the fact that 
causal relationships between classroom instruction and learning 
outcomes are erroneousl> presumed b> current evaluation practice 
further constrains the utilization- of evaluation findings. Given this 
contextual background, O'Shea argues that it is reasonable to 
expect that current evaluation practice will provide information 
having onl> limited fmpact on^the wa> in whicH school districts 
make internal instructional decisions. He concludes with some rec- 
ommendations for overcoming -this problem. 

This monograph, and the CSETwork out of which it grew, is 
unique. The project brought together a number of sociologists w ho 
first contributed to the design of^empirical surve> work being done 
on school district evaluation practice and next reflected upon the 
findings from that work from the standpoint of theories and -liter- 
atures not common to educational^valuators. 

The survey was initiated because CSE recognized that educa- 
*tional evaluation, as it occurred within school districts, was vir- 
tually undocumented b> researchers. Although man> school 
districts were themselves conducting, or employing others to con- 
duct, evaluations in response to federal and state reporting 
requirements, ver> Httle summary information was available about 
how they were managing their evaluation tasks. CSE was aware 
that some of the larger, and a few smaller, districts had cerltralized 
their evtJuation functions into rese^arch and evaluation units. Both 
CSE and the National Institute of Education were curious about 
ho^w^ these evaluation units were staffed, funded, and organized. 
Consequently, CSE was contracted by NIE to investigate these 
issues at the national level. ' 

The survey was intended, primarily, to describe the centralized 
evaluation capacity being developed within districts. But it seemed 
to us that the organizational characteristics of school district central 
offices, as well as the organizational characteristics pf evaluation 
units themselves, merited additional scholarly attention from spe- 
cialists \|^o pQuld describe the surrounding confext and weigh its 
influence on evahiation practice and usefulne'iss. Consequently, we 
organized a team of individuals to represent both viewpoints. 

The early meetings of this team were devoted to broad discus- 
sions of research strategy and to speciiic items that should be 
included in the questionnaire seeking information about school 
district evaluation units. The bulk of the responsibility and day-to- 
da> work involved in designing and administering the question- 
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naire and mitidllv analyzing its results was p*erformed under the 
supervision of Catherine Dillon L>on, who at th^t time was project 
director. During the formulation of the survey the contributions 
of the team were primarily supplementary and supportive. Late^:, 
after CSE project staff had prepared a descriptive study from the 
survey responses (Lyon, et al.,- 1978, Williams, 1979), four mem- 
bers of the team— the four chapter contributors in this mono- 
graph — undertook their own independent analyses of the data. 
What was unique about our joint CSE-specialist efforts, then, was' 
, their interdiscipUjiary nature. ./ 

In the remainder of this introduction Ve briefly review some 
recent organizational perspectives for understanding school district 
ftmctioning, as well as recent evaluation perspectives relating to 
issues of utilization. Then we describe the CSt survey of school 
district evaluation units and supimarize its major findings. ^ — ^ 

In the body of the text, we precede each chapter with a short 
abstract summarizing the major arguments and their significance 
The volume concludes with a synthesis and brief discussion of the 
implications of this work. * 

4 

Recent Perspectives on School Districts as Organizations 

The understanding of organizations and of administrative deci- 
sion making has undergone important change? in reeent decades 
Earlier, the dominant, idealized view of organizations was that of 
a rational, clearly designed bureaucratic structure staflFed by workers 
whose main preoccupation was the organization's successful func- 
tioning. Other organizational manifestations, such asjnformal 
structures, staff self-interest, decision maker prjeferences, favor- 
itism, and the like were considered deviations frbm the ideal and 
summarily described as system "noise." The job of the good man- 
ager was to reduce that "noise" and keep the organization running 
smoothly, vC^ithout irrational interference. Thus, managers were 
taught that they would be successful if their organizations were 
properly organized and managed— using such concepts as span of 
control, specified goals, and clearlv written rules and regulations. 
Writings in the educationa] acfministration literature clearly 
reflectad this dominant organizational viewpoint. 

Over the years, however, other perspectives on organization 
have been gaining acceptance. What was previously viewed as 
irrational, e.g., the dominance of administrator preference in 
decision making, or goal displacement, or informal subsystems 
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became increasingly viewed as natural outcomes of organized 
human endeavors. Organizational theorists have sought to embod> 
such concepts and activities within their thinking and formulations. 
Another wa> of describing this phenomenon is to consider the field 
as moving from a relatively simplistfc certainty to a relatively 
complex uncertainty. But as of now, there appears to be no all- 
encompassing organiza^bhal theory that adequately captures the 
complexity of organte^ional functioning. 

Ir this monograph the authors refer to a wide range of notibns 
an4 concepts about organizations, and the reader iinacquainted 
with the organizational literature may not be familiar with them. 
Among the more important of those concepts are "loose coupling" 
(Weick, 1976), "institutional' vs. "technical" organizations (Meyer 
& Rowan, 1977), and "organized anarchies" (March & Olson, 
1976). ^ 

Loose coupling refers to the degree to which various parts of an 
organization (its individual staff, divisions, units) are linked to one 
another. If management acts to establish a new employ ee proce- 
dure, for instance, will it be implemented at the employee level? 
If not, that part of the organization can be considered to be loosely 
• coupled. If the desired effect is quite immediate, it is tightly cou- 
pled. For example, a superintendent might require that all teachers 
use certain materials in their classrooms. If, after a reasonable 
period of time, his or her preferences were not routinely imjjj^e- 
mented, the district s functioning as regards that specific order 
would be considered loosely coupled. It should be noted that 
organizations can be simultaneously loosely coupled in some mat- 
ters and tightly coupled in others. 

Institutional and technical organizations differ with regard to 
^ whether they primarily articulate societal norms ^r^a specific tech- 
nology. Technical organizations, such as manufacturing enter- 
prises, implement a specific technology . Since management has 
full knowledge of all processes required to transform raw material 
inputs into finished product outputs, the organizational structure 
can be tightly coupled. If a technical organization has an evaluation 
unit, that unit can relate deficiencies in quality of output back to 
the specific production processes responsible. 

In contrast to technical organizations, institutional organizations 
implement societal expectations in areas such as religion, welfare, 
and education. These provide service in areas that the community 
wants addressed, but there is a limited body of tested knowledge 
upon which the managers of those organizations can draw to 

^ « 
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implement procedures guaranteed to produce desired objectives. 
Implementation, therefore, commonly is delegated to certified* 
professional practitioners whose activities are^nl> loosel> coupled 
to management. Neither managers nor evaluators u$ualj> attend 
directly to the central technical problems but rather deal with 
them indirectly or symbolically. School district administrators, for 
example, may define a good school as fulfilling social expectations 
regarding credentialled teachers, requisite numbers of library 
books, and a publi^ed instructional continuum. How the creden- 
tialled teachers actually teach the instructional continuum is dealt 
with only incidentally. An evaluation unit may prefer to devote its 
resources to demonstrating compliance to external expectations 
rather than to assessing the effectiveness of instructional proce- 
dures relative to pupil achievement outcomes. 

Organized anarchies is a term used to describe phenomena 
actually occurring in many organizations. These phenomena are 
quite counter to what rational bureaucratic theory would lead one 
to expect to find. The term attempts to portray something that may 
be organized but may not be rational according to standard orga- 
nizational theory. Proponents of this view (fpr example, March & 
Olson, 1976) argue that many organizations, either occasionally or 
usually, exhibit this kind of behavior to a lesser or greater degree. 
The behavior can be described as "choices looking for problems," 
^'solutions looking for issues," and "decision makers lookirjg for 
work." 

These concepts — loose coupling, institutional organization, and 
organized anarchies — together suggest quite a different view of 
organizations than is pictured by rational bureaucratic models. But 
these three concepts do not describe organizational aberrations, 
rather they represent the organizational world as it really is. 

The extent to which school districts operate as loosely coupled, 
institutional, and anarchic organizations has profound implications 
for thinking about and perhaps re-considering the role and value 
of educational evaluations in districts' decision making. At the very 
least, it probably means that evaluators have to come to under- 
stand the organizational constraints within which educational 
administrators make decisions. This understanding is crucial jf- 
evaluation information is intended to be used in the districts' 
decision-making process. 
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Recent perspectives on Evaluation 

Within the past five years, as evaluation activities have been 
increasing!) mandated at policy levels and conducted at local . 
levels, evaluation scholars have been concerned about utilization. 
Some scholars examining evaluation utilization view the phenom- 
enon or its absence as a special case of the study of knowledge 
utilization (Caplan, 1975; Weiss, 1979, Boruch, 1980). They are 
concerned with how the production of research knowledge in spe- 
cialized fields, including education, gets transmitted in a useable 
form to those who must make policy decisions. The audiences of 
principal concern to these researchers are primarily, though not 
^ exclusively, federal and state legislators and administrators who 
must allocate scarce resources among competing program 
alternatives. 

There are also individuals working on evaluation utilization from 
the point of view of defining the problems inherent in the practice 
of evaluation and improving its procedures and practices (Alkin, 
Daillak, & White, 1979, Patton, 1978). These researchers view 
evaluation as a service to decision makers, and reason that if such 
services are not perceived as useful to clients — at whatever level 
the clients may be — then the evaluation profession needs to reex- 
amine the way in which it goes about its work. 

A third group (Apling, Kennedy, & Newman, 1980, Thompson 
& King, 1981, Williams & Bank, 1981) approaches evaluation and 
th^ data it generates as one of many sources of information residyig 
within a dynan;iic and often politically volatile organizational con- 
text. They examine evaluation as one among many ongoing pro- 
cesses and they consider the use or non-use of evaluation findings 
in local districts in terms of relationships among those processes. 

It is, only within the past five years that this last group, which 
is interested in examining the organizational (as distinct from the 
knowledge production or the professional) aspects of evaluation 
utilization, has become a major force within the evaluation research 
communi-ty. We believe that CSE's work, underwritten by NIE 
and expressed in this monograph, is an important contribution to 
this development. The monograph depicts the factors affecting 
organizational decision making and describes some of the tensions 
existing among actors within an organization. Since educational 
decision making usually does not follow the evaluator s rational 
I model of problem statement, alternative solution generation and 

weighting, followed by the selection of an optimum solution, the 
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monograph provides some tentative advice for both evaluators and 
decision makers which ma> help them recognize each others' con- 
straints and mutually work towards increasing the usefulness of 
evaluation information. 

The CSE National Survey 

The 'national survey was undertaken to broaden our under- 
standing of how evaluation activities contribute to the renewal of 
public education at the school district level. Its primary function 
was descriptive. As an indication of the lack of basic information 
about school district evaluation functions, it took CSE^staflF more 
than six months simpl> to locate the appropriate respondents* 
ns(Tfi?5 and addresses and generate a basic mailing list. 

Survey respondents were finally selected through a two-stage 
process. First, letters were sent in the fall of 1978 to all 750 super- 
intendents in school districts with 10,000 or more students and to 
a 50 percent sample of the 573 school districts with 5,000-9^^999 
students. All of the larger districts an<^81 percent of the smttller 
ones responded indicating whether or not their district had an 
evaluation unit. Next, in spring, 1979, a questionnaire was sent to 
the 320 large school districts (10,000 or more students) and to the 
74 smaller ones identified as having evaluation units. A total of 263 
unit heads (or 64 percent) returned the survey questionnaire. 

The findings include the following:^ 

Only 43 percent of the districts enrolling 10,000+ have a central 
office responsible for program evaluation. The existence of an eval- 
uation office is partially a function o^ district size. Almost all (89 
percent) of the metropolitan districts, 59 percent of the large, and 
33 percent of the medium districts have^n evaluation unit. 

Formal school district evaluation is a relatively young and growing 
field in education. Only 15 percent of the existing evaluation offices 
were established before 1965. Since 1970 the creation of these 
offices has, accelerated, with 51 percent newly organized or reor- 
ganized from other units (testing, research) since then. 

Generally, ev^uation offices are mor^ Jikely to be in one of the 
typical lines of authority, rather than directly reporting to the 



*From Lyon and Doscher (1979). See also Lyon,^ Doscher, McGranahan & 
Willians (1978) and Williams (1979). . 
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Superiiftendenl:. Thirty-seven percent of the res^pondents report 
directl) to the Superintendent, 44 percent report through an inter- 
mediar), and about 20 percent through two or more intermediaries. 
Thirty-nihe percent are located in instruction offices, leaving 61 
percent located elsewhere (administration and support services, for 
example). 

Central evaluation offices are primarily though not likely to be 
exclusively in charge of formal assessment. Only 26 percent report 
exclusive responsibility for evaluating both state/federal and local 
programs. . 

Evaluation office activities are dominated by school dtstrict per- 
sonnel rather than consultants. Almost half of the units do not spend 
any money on consultants. Two-thirds of the respondents reported 
that their personnel resources are not adequate. . 

Most evaluatiori offices administer testing programs.. .Seventy -one 
percent administer district-wide norm-referenced , testing (NRT) 
programs. Forty -eight percent administer district-wide criterion- 
referenced testing (CRT) programs. Districts administering both 
CRT and NRT programs use them for the same purposes. Approx- 
imately 75 percent agree that testing is their major method of data 
collection. ' 

The average ^valuation office spends approximately 40 percent of 
its time on instructional clients compared with 60 percent with 
administrators. Generally, administrative rather than instructional 
personnel are perceived to be using evaluation unit data. Evaluation 
offices typically do not take part in other administrative functions 
(personnel, collective bargaining, budget). Offices express little 
, agreement on what constitutes basic evaluation practices or on the 
priority of various evaluation activities. 

Conclusion 

The preceding pages only touch upon the thematic concerns of 
the monograph. Each of the topics we have raised is amplified in 
one or more of the following chapters. Together, the authors gen- 
"^erate theoretical propositions about the role of evaluation in edu- 
cational orgartTzations, test some of these propositions against CSE's 
empirical survey data, and offer valuable insights to both decision 
maker and evaluator. While we do not claim to have illuminated 
a direct path to the heart of the labyrinth, perhaps we have sug- 
gested the dimension and complexity of the maze and indicated 
some of the promising directions to take. 
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Notes on O'Reilly's 
"Utilization of Evaluation Research'' 



Introdqcdon 

In the 1960*s, the early years of educational evaluation, much 
attention was directed, quite properly, toward evaluation's tech- 
nical aspects. The field was dominated by such questions a^. What 
^ is an appropriate design to assess the effectiveness of an educa- 
tional program? What stafistical.methods are most useful to analyze 
complex multi-level data? How do you write clear, measurable 
objectives that reflect the intention of the program developer? 
What are effective data collection techniques? Some of these 
emphases can be traced back to the origins of the evaluation move*- 
ment, some can be traced to the interests of the psychologists and 
^ measurement specialists who provided the movement's *early 

leadership. 

Considerable progress has been made toward resolving these 
methodological concerns, although there remains much healthy 
debate in the field over what methods are most appropriate under 
what^circumstances. 

iMore recently, some evaluators have begun shifting their atten- 
* tion to a*different set of questions which have to do with the uti- 

lization of evaluation data. The kinds of questions being asked are. 
Who will ever read this evaluation report? Are these evaluations 
to be used by decision makers and, if so, under what circumstances 
and for what purposes? What factors inhibit or increase evaluation 
, use? Research on evaluation utilization has provided us with mixed 
ans^wers. Evaluation reports are being used in some instances and 
not in others, often their use is as one source of da.ta that fits into 
thp decision maker^* thinking, in some instances evaluations can 
be decisive; in many, they have more limited influence. 
^ Those interested in trying to better understand evaluation uti- 
lization and there^^y provitte guidance to those seeking to improve 
the level of utilization, have worked in different, but'complemen- 
tar> 'directions. Some view evaluation utilization as a subset of 
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knowledge utilization and so examine the roles of information pro- 
duction and transmission. Others have sought answers through fur-* 
ther improving the technical and methodological side of evaluation: 
Thev reason that decision makers' non-utilization res\s in part on 
their perception that evaluations are not technicall> slpund or are 
poorl> written or poorl> timed. Others have begun t6 look more 
closel> at the decision-makir\g context and the characteristics bf 
decision makers and the organizations they work in. The> ask such 
questions as. What are the man> things decision makers consider 
when reaching a decision? What personal or organizati6nal or 
environmental constraints influence decision^making behavigr? 
The study of decision making is, of course, not a new field of . 
^inquiry^.Eor decadev-indeed_centuries, scholars and practitioners 
alike have been fascinated with decision making and decision 
makers. That Herbert Simon won the Nobel Prize for his work in 
this field attests to its importance. Because of its obv ious relevance 
to administrators and managers, much of the work has originated 
in and been encouraged b> professional schools of administration 
and management. 



Argument 

Charles O'Reilly III, a sociologist in the Graduate School of 
Business Administration, Universit>' of California, Berkeley, is a 
student of organizations and decision making. In his chapter he 
attempts the difficult task of selectivel> reviewing several relevant 
literatures— sociology, social-psychology, decision making, admin- 
istrative Jtheor>, organizational theory . He then draws out propo- 
sitions that will provide insights to those interested in the evaluation 
utilization problem. 

The main thesis of his argument is that organizational 'leaders 
and decision makers are not neutral entities who are waiting for, 
or .even desirous of receiving, carefully constructed evaluation 
studies that objectively report and interpret data that will help 
them reach informed decisions. Instead, organizational leaders and 
groups in organizations have personal, political, social, and eco- 
nomic influences and constraints that largely determine the deci- 
sions they make. Evaluation may play a minor, and occasionally 
a major role in decision making, especially if the findings are con- 
sistent with what the decision makers had already concluded was 
the best course of action. 
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0*Reill> reminds hs that decision making in complex organiza- 
tions, such as school districts, is not usually the result of one single 
event nor the product of one individual. Rather, it evolves from 
complex and ongoing bargaining and negotiation, both formal ancl 
informal. Information, as represented bv evaluation, is sometimes 
ammunition in a continuing battle between those factions that sup- 
port or oppose a given decision. Seen in that light, it is easier to 
understand ^vhy evaluations can be valued or discredited, quoted 
or ignored, regardless of their intrinsic technical characteristics. 

Contribution and-Imgortance * . ^ 

To readers noi familiar with this line of inquiry, approaching 
O'Reilly 's chapter ma> seem a formidable ta^k. The language, the 
approach, and the format ma> be unfamiliar. But his work makes 
several important contributions to the evaluation utilization ques- 
tion. He introduces the reader to the major research in several 
related fields, he spells out the c6fnplexit> of the utilization problem 
and thereby reduces the likelihood that the evaluation field will 
reachssimplistic solutions to complex problems, and finally, he pre- 
sents a series of tentative propositions that evaluation scholars and 
practitioners alike should consider when conducting evaluations or 
inquiring further into evaluation utilization. 
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Evaluation Information and Decision 

Making in Organizations: 
Some. Constraints on the Utilization of 
Evaluation Research^ 

Charles O'Reilly III 
University of California, Berkeley 



INTRODUCTION 

The CSE project studying evaluation activities in school districts 
addresses the question of the extent to which school district eval- 
uation offices are contributing to the renewal of public education. 
Part of this question can be answered empirically By examining 
data describing how local evaluation agencies (LEA's) are orga- 
nized and staffed and what functions they serve (Lyon, Doscher, 
McGranahan, & Williams, 1978). Still another part of the question 
can be addressed through organizational analyses using survey data 
to test propositipns concerning the impact of structure or conflict 
(see Grusl<y and Zucker elsewhere in this monograph). As David 
(1978) has documented, it appears that Title I evaluation resukS 
have not served, primarily, as a means of judging program effec- 
tiveness or as a guide to program improvement. It is also apparent 
that evaluation information is rarely'seen as important in the con-* 
tej(tt of decisions about program change?. David goes on to point 
out that "literature on evaluation has only recently included 
attempts to understand the role and use of evaluation results ... in 
the realm of decision making" (1978, p. 8). Her study provides 
evidence of the almost total lack of utilization of evaluation data by 
decision makers in school districts. 

Since the CSE survey data do not directly address the empirical 
testing of the propositions generated here, this chapter may best 
be seen as an attempt to make' the reader, awarq of some factors 



» I would like to thank John C. Anderson, Susan Resnick, and Bill Click for 
helpful comments on an earlier version of this manuscript 
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which ma> preclude evaluation information as a force for change. 
From this perspective, local evaluators ma> be able to anticipate 
what aspects of their evaluation will be seen as convincing or useful* 
by various decision-making audiences. ^ 

BACKGROUND 

In the past several years, evaluation researchers and practi- 
tioners have become concerned about the apparent non-utilization 
of ev aluation information. For instance, Davis and Salasin (1975) 
acknowledge fhat even methodologicall> sound research is often 
not used. Because many people accept the proposition that the 
purpose of evaluation research is to produce information primarily 
for use b> decision makers, their concern about decision makers* 
non-utilization of evaluation findings calls into question the ver> 
purpose of evaluation. 

There is^no true consensus on whether this perceived lack of 
utilization is accurate. There is evidence supporting and evidence 
attacking the claim of non-utilization. For exan;iple, a number of 
authors have noted that the Head Start program continues, in spite 
of the results of ten or so years of research demonstrating that it 
has had almost no eJQPect on cognitive abilities (e.g., Goldstein, 
Marcus, & Rausch, 1978). Similar^in^tances of what appears to be 
non*utilization can also be founiSTVVard and Kassebaum (1972), for 
example, report that program administrators in a state correctional 
system *s counseling program systematically ignored evaluation data 
showing that the program was ineffectual with respect to the spec- 
ified outcomes, decided to extend the program and, in certain 
cases, to make it mandatory. 

A number of reasons have been offered in explanation of such 
non-utilization: 

• A lack of rigor in the desjgn and conduct of the evaluation 
study (e.g., ^Goldstein et al., 1978). 

• So much rigor that the research fails to <issess the true pur- 
pose of the program (e.g.. Cox, 1977). 

• Reporting outcomes which are not helpful (such as negative 
findings), non-timdy reporting of results, or providing reports 
which are too long anc} filled with jargon (Alkin, 1975; Cox, 

. 1977; Goldstein et aL, 1978). 

• Communication problems between evaluators and program 
administrators (Cox, 1977; Goldstein et al., 1978). 

25 
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^ Arraved against these arguments and ^^^nples are counter- 
examples ^mci claims that utiliz<itipn does, in fact Occur. In a surv ey 
of federal evaluators, Patton (1978) reported that 78 percent of 
.those survived felt that evaluation studies had an impact on their 
programs. He. and others, argue that a broader definition of "uti- 
lization" must be used in order properly to assess the impact of 
evaluation research. When the definition of ut-rtnation is extended 
to include impacts such as providing additional credibility for a 
program, verifying managers' suspicions, and generally providing 
additional information, then, it is argued, the utiliz<ition of eval- 
uation mformation is more widespread than is commonly believed 
' (Lev ine & Levine, 1977). The problem, according to tliis view, is 
priniiiriK that the definition of "utilization" is too narrow and 
^ Qverly rational" (Patton, '1978,^ p. 40). 

Whether utiliziition is or is not problematic depends on the-def- 
. inition of the term, on what is meant by "rational," and on the 
evidence the discussant uses in arriving at a conclusion, on what 
information the discussant s^ks out oris exposed to, how closely 
he or she chooses to attend taithe various arguments, and whether 
he or she has a vested interest in the outcome of the dispute. From 
this perspective, the utiliziition or non-utilization of evaluation 
information d<?pends on information availability and use as well as 
the organiziitional processes and pressures which encourage or 
discourage the production, transmission, ifhd use of information by 
relev ant decision makers. 

This chapter analyzes the utilization of evaluation research from 
an ()rganiz<iti()nal behavioral perspective. To accomplish thijK I will 
^ present a simplified information-processing model of decision 
making. With this orientation, the utilization or non-utilization of 
evaluation i^f()^mation may be seen as a function of two primary- 
factors*. (1) the .orgcini/iitional context in which decision makers 
function, including an awareness of operating goals and incentives, 
and (2) the information processing constraints on the decision 
makers, including availability of information, costs of search, and 
the manner in which information is perceived and processed. 
Drawing on tlie research literature related to information u^)e and 
organiziitional decision making, a series of propositions will be gen- 
efjited that suggest conditions under which evaluation information 
is likely to be used, or not used, by deci.sion makers. 
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AN INFORMATION PROCESSING MODEL 
FOR DECISION MAKING 

Decision ina^ng is, simpl> put, t^act of choosing amon^ alter- 
natives. Ideally, the decision makeFre presumed to have a set of 
values or evaluative criteria, the perception of a problem which 
requires action, a number of potential alternative solutions, and 
a calculus for comparing alternatives and estimating the likelihood 
of attaining certain outcomes given certain alternatives. With per- 
Jed rationality , the assumptions leading to choice generally include 
complete information about the alternatives, knowledge of the 
probabilities associated with diflFerent alternative-outcome links, 
a consistent ordering of preference among outcomes, and a selec- 
tion mechanism which maximizes the value attained by a choice. 
However, with bounded rationality, the limits on both the infor- 
mation and- cognitive processing abilities, of the decision maker are 
recognized, that is, the decision maker is intendedly rational but 
acting with Ifmited computational abilities. In both cases, hov^f- 
ever, the choice process" is one of information assimilation and use, 
suggesting that an information processing model ma> be a fruitful 
way to examine the deqision-making process. 

Consider the simplified model of decision making presented in 
Figure 1. In this schema, the decision maker faces a problem or 
situation requiring a choice, generates potential alternative solu* 
tions, assesses the probabilities that a given alternativq will lead 
to certain outcomes, and develops a preference ordering among 
outcomes. In this view, information and the abilit>^ to process it 
are paramount. Information processing is required in all phases, 
i.e., to define the problem, develop alternatives, estimate proba- 
bilities, and order outcomes. 

The emphasis in this model is not that^it is a complete or litjeral 
representation of the decision-making process. A variet> of similar 
models have been proposed (e.g., Cohen, March, & Olsen, 1972; 
Janis & Mann, 1977, MacCrimmon, 1974), and there is evidence 
suggesting that such rnodels may be oversimplifications (Witte, 
1972). Mintzberg, Raisinghapni, and Theoret (1976), for example, 
traced 25 complex organizational decisions anJ characterize the 
process as a pluralit> of sub-decisiorts without a simple sequential 
relationship such as indicated in Figure 1. Nevertheless, the model 
in Figure 1 is useful in that it emphasizes the centrality of infor- 
mation and information processing in the decision process. It also 
' suggests a framework for organizing and investigating the limita-^ 
tiqns and constraints on organizational decision making. 
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In a t>pical formulation, decision makers encounter or are pre- 
sented with problems which require choices, e.g., continue funding 
for a project or reallocate funds to other purposes, or allow the 
hiring of additional staff. The decision maker then considers a 
variety of alternatives for solv ing the problem. Each alternative is 
examined and some subjective weight or probability estimated that 
selection of given alternatives will lead to particular outcomes. 
T>picall>, the decision maker is no^ indifierent among the set of 
outcomes but has preferences reflected b> weights for at least some 
of the outcomes. Given that information processing limits exist, it 
ma> be that the decision maker will not have complete knowledge 
of the alternatives, probabilities, or outcomes. In general, how- 
ever, the decision-making process is presumed to operate from left 
to right. Under the traditional notions of rationality ascribed to b> 
evaluation researchers, it is assumed that the decision maker will 
search for unbiased information about the various components in 
the model, and that the weights attached to various outcomes are 
determined b> organizational goals, that is, the decision maker will 
attempt to make the choice such that the maximum net benefit 
accrues to the organization or the agency. % , , 
, But is this left-to-right progression an accurate description of 
how decisions are made? Several authors have projJosed that, in 
organisations, problems are seldom clearly defined and alterna- 
tives become known only after certain outcomes are preferred 
(Cohen et al.,'l972; Weick, 1977), It may be, in fact, that the 
decision-making process in organizations begins with the prefer- 
ence-ordering for outcomes as a set of rather fixed constraints. 
Simon (1964) suggested this when he proposed that organizational 
goals be viewed as constraint sets. Certainly, in organizations 
individuals and ^ubunits typically have vested interests. The loss 
of resources, status, and power is to be avoided. Even the uncer- 
tainty caused by change may be resisted. On th^ other hand, 
increases in resources, for example, more funding or more staff, 
are typically preferred. Decision makers are indifferent about out- 
comes only when they are unaffected* In fact, decision makers may 
become participants in a decision-making process to insure that 
they remain unaffected. 

That organizational decision makers have strong preferences for 
certain outcomes is well documented (e.g.i* Pfeffer & Salancik, 
1977, 1978). These preferences, discussed at greater lengtl] later 
in this chapter, reflect not only organizational and subunit goals, 
but also individual desires and concerns such as promotion and 
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salarv . Because these preferences exist, it follows that managers 
or administrators are unlikely to remain passive when decisions 
affecting them are to be made. Conflicts mav develop when deci- 
. sJons can result in gains and losses to the participants. Under these 
circumstances, alternatives which lead to undesired outcomes ma> 
become unacceptable to certain participants. In addition, as 
Thompfjon (1967) observed, uncertainty threatens rationality. 
Managers, especialh those in power, attempt to eliminate poten- 
tiallv disruptive uncertainty. Information, in this context, becomes 
a politiccd resource. Data which support desired outcomes are 
sought out, while information which supports opposite views is to 
<be rebutted by questioning its accuracy, for example, by obtaining 
other information supporting a countervailing view, or by 
mpugning the credibility of the source of the threatening infor- 
mation. Information, then, is a commodity used for a variety of 
purpo.ses. Under some circumstances it may be used as a basis for 
decision making, in others, as corroboration for decisions already 
made, and in still others, for symbolic reasons. Information, in this 
miheu, is not a fixedj^ubstance, but one which may be selectively 
perceived and processed. 

Thus, the argument being made is twofold. Firsts in organiza- 
T» tions, participants typically have preferences for outcomes which 

reflect organizational as well as individual goals. These 9utcomes 
act to define a set of constraints which result in the decision process 
moving from right to left with reference to Figure 1. Second, 
because of the potential for disagreements among participants, the 
entire decision process may be one of bargaining and negotiation 
as various actors attempt to pursue their interests. In this political 
process, information becomes a potentially important, or threat- 
ening, commodity. It is generally not perceived of as "objective." 
Rather, depending on the nature and importance of the goals 
sought, decision makers may systematically search for supporting 
information while ignoring other types. 

It is this organizational context in which evaluation research and 
.evaluation units, whether at local, state, (5t federal level should be 
examined. Both the evaluation unit and the information it pro- 
duces may be assets or liabilities for decision makers. 

A further distinction reflecting categories of evaluation infor- 
mation IS necessary for the purposes of this chapter. For example, 
one can distinguish between formative and summative evaluation, 
with the former referring to information focused on improving the 
workings of a given program and the latter more concerned with 
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its overall impact or effectiveness. Formative fevaluations are typ- 
ically used to provideanformation to administrators tg improve the 
conduct of the program and, as such, ma> be less threatening than 
a summative evaluation which ma> dictate the continuation or dis- 
continuation of the entire project. We can make still another dis- 
tmction between information collected for routine compliance with 
funding agency guidelines and non*routine information used to 
assess goal attainment. The former information is often critical in 
that continued funding is predicated on demonstrated compliance 
and substantial efforts and resources are often required ^to collect 
such information. This information, however critical, is not typi- 
cally used to assess progranri performance. For the purposes of this 
chapter, "evaluation information" or "evaluation research" wall 
refer primarily to the more non-routine, summative aspects of 
evaluation research. While some of the material developed may 
be relevant to an understanding of routine or formative evaluation, 
the primary focus here is on information about non-routine or sum- 
mative events. 

The remainder . of the chapter elaborates these views and 
develops a series of propositions concerning circumstances under 
which evaluation units and information are likely to be effective. 
First, the concept of "rationality" as it applies to organizational 
decision making and information use is discussed. Second, the 
impact of organizational goals and incentiv es on the model present 
in Figure 1 are considered. Finally, drawing upon a diverse body 
of research on cognitive information processing and decision 
making, a number of potential biases in information processing 
which may affect the use of evaluation information are explored. 

RATIONALITY AND THE DECISION-MAKING PROCESS 

As indicated previously, the notion of rationality implied by- 
Figure 1 is not the comprehensive rationality of economic theory^ 
in which unbiased decision makers use perfect information to max- 
imize utility according to some completely specified and ordered 
pr^erence set. Instead, decision makers begin With preferences 
and select actions base'^.on imperfect expectations about their 
effect upon future preferences. These actions, further, may include 
limited and focused information search and selective percepti&R 
and processing of information. Thus, "rationality," as used here, 
does not even refer solely to the notions of bounded rationality in 
which choices are made by decision makers who use imperfect 
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information in a satisfying manner to maximize goal attainment * 
(March & Simon, 1958). Instead, the appropriate calculus for 
Figure 1 appears to be one which March (1978, p. 592) refers to 
as "contextual rationalit>" in which choice behavior is embedded 
in a complex of othfer claims on the attention of decision makers 
and other structures of social and cognitive relations. Organiza- 
tional decision makers, in this view, are pursuing multiple objec- 
tives subject to a variety of pressures and constraints, and often 
with considerable ambiguity surrounding the choice process. Under 
these circumstances, preferences for outcomes ma> be the least 
' ambiguous cooiponent of the decision process, more certain than 
the definition of the problem, the range of feasible alternatives, or 
the probabilities associated with various alternatives. In this situ- 
ation, it is argued that decision makers are likely to take action 
which both reduces their uncertainty and helps them achieve 
desired outcomes (e.g., search for supportive information or selec- 
tively interpret signals as favorable to a preferred outcome). Lind- 
blom (1959) offers some support for this in his observation that the 
selection of goals and the empirical analysis of the needed^ctions 
to obtain the goals are not distinct from one another but are closely 
intertwined. Since ends t>picall> come before means and ma'y be 
known with greater cerf^nty, it is likely that the search for appro- 
priate means will be highly focused. 

Since individual decision makers are known to be limited in 
their ability to solve problems, organizational routines are often 
established to increase the likelihood that individuals will behave 
in a traditionally rational way. Tfcis focus is on the effectiveness of 
procedures used to make choices, what Simon refers to as "pro- 
cedural mtionality" (1978, p. 8). These routines are established, 
in light of the limitation^ suggested by bounded and contextual 
rationality, to emphasize rational search procedures and, insofar 
as possible, to ensure that decision majcers have complete infor- 
mation. This procedural rationality may easily include provisions 
for information and control systems, carefully^rescribed review 
processes, and even mandated program evaluations. The explicit 
attempt is to ensure that the organizational context promotes an 
approximation to comprehensive rationality by decision makers. 
These manifestations of procedural rationality may be of consid- ^ 
erable symbolic importance (e.g., Meyer & Rowan, 1977), but of 
limited* efficacy if decision makers are only contextually rational. 

Such a reformulation of traditional notions of rationality^ is hardly 
new. For a number of years economists, political scientists, and 
others have acknowledged that "perfect rationality" is not an apt 
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description of real-world decision making (e.g., Etzioni, 1967; 
Floden & Weiner, 1978, Gershuny, 1978, Lindblom, 1959). What 
is, perhaps, relevant here is that some of the disillusionment with 
social science evaluation can be partly attributed to at least an 
implicit acceptance of the assumptions underlying traditional ratio- 
nality. It is worth re-emphasizing that in an organizational context, 
unlike what is implied in the traditional rational model, goals are 
often ill-specified or lack consensus, information may be incom- 
plete, ambiguous, and imperfect, and decision makers may be pur- 
suing multiple competing objectives and lacking the tin^e and 
computational abilities necessary to adequately utilize the available 
information. It may even be, as suggested here, that the decision 
itself is driven by the solution rather than the problem. Acknowl- 
edging these constraints, let us now turn our attention to. (1) some 
of the relevant organizational properties alluded to by the notion 
of contextual rationality , and (2) the cognitive processing limita- 
tions suggested by the notion of bounded rationality. 

ORGANIZATIONAL IMPACTS ON DECISION MAKING 

Organizational Power and Evaluation 

To understand the impact of contextual influences on evaluation 
research, let us" first consider the evaluation unit in an organiza- 
tional context, that is, consider the evaluation unit as one group 
among many subunits. As Pfeffer and Salancik (1978) have noted, 
subunits within organizations may be viewed as actors competing 
for resources in a political arena. Power, in this milieu, becomes 
a critical determinant of the unit's ability^ to acquire scarce and 
critical resources (Pfeffer & Salancik, 1974). 

Power, the crucial ingredient in this contest, may be envisioned 
as resulting from the subunit's ability to control critical contingen- 
cies, that is, power stems from the ability to reduce the primary 
uncertainty which feces the organization. As formulated by Hickson, 
* et al. (1971), power or the ability to control contingencies and 
reduce uncertainty may vary according to. (1) how critical the 
uncertainty is which the subunit can reduce, that is, how central 
the Contingencies are for the overall functioning of the organiza- 
tion, (2) how effective the subunit is in reducing the uncertainty, 
and (3) how "substitutable" the function served by the subunit is, 
that is, how easily the uncertainty reduction can be obtained from 
other sources within the organization. Subunits that are successful 
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according to these criterid are likely to be powerful. Those units 
that do not act to reduce important uhcertaint> for the organiza- 
tion, or whose ability to reduce uncertainty is not uniqudy held, 
are unlikely to exert substantial influence within the organization. 

Subunit power has important ramifications for an understanding 
of the utilization^ of evaluation information for several reasons. 
First, PfefFer and Salancik (1974) have shown that ppwer is directly 
related to a subunit s ability to obtain scarce and critical resources, 
for example, budget allocations and staflF. Without these, it may be 
difficult for an evaluation unit to perform effective evaluations. 
Second, Pfeffer and Salancik (1977) have also demonstrated ^hat 
power is used to define the criteria used in decision making. More 
powerful subunits may successfully emphasize those criteria on 
. which they compare favorably. This ability to define or specify 
evaluative criteria may have direct impacts on program evaluation, 
allowing powerful programs either to specify' the basis on which 
they are to be evaluated and thereby insure their continued suc- 
cess, or to invalidate unfavorable findings by redefining posteriori 
the evaluative criteria. Without power, members of the evaluation 
unit are likely to find that they lack discretion in selecting the cri- 
teria and rejecting others' definitions (e.g., Meltsner, 1976), or that 
they adhere to the more universalistic standards of research only 
to have their product labelled as irrelevant or ignored by decision 
makers. 

A final ramification of power which has been noted is the ten- 
dency of those in charge to institutionalize their power and to resist 
changes which might alter their position. Goldstein, et al. (1978) 
describe, how groups often desire evaluation research to satisfy 
external demands to "do something, ** but simultaneously are looking 
for the results to justify established policies and procedures. 
Decision makers, it seems, are more receptive to research conclu- 
sions that fit nicely into established policies. This tendency — and 
ability — of powerful units to maintain their position acts to slow 
down the process of change and may easily contribute to both a 
desire and an ability on the part of powerful subunits to attend to 
selectively (or ignore) utilization information, depending on their 
interests.' 

These findings suggest several propositions with respect to the 
influence of evaluation units and the utilization of evaluation 
research: 
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Proposition 1. Results of evaluation, research are more likely to be 
used by decision makers when: 

a. The evaluation unit is powerful (Compared to the unit being 
evaluated. 

. b. the evaluation information is not available from other sources, 
i.e., the control of uncertainty is not substiti^table. 

c. the evaluation study is on a program requiring a substantial pro- 
portion of the agency's resources, i.e., the project evaluated is 
central to the agency's functioning. 

d. the criteria used for evaluation are criteria which the unit being 
evaluated has accepted, i.e., there is agreement between the 
evaluating unit and the one being evaluated on what is to be 
evaluated. 



Organizational Coals, Incentives, and Evaluation 

If; as suggested in Figure 1, the use of information may be a 
function of the outcomes preferred by a decision maker, it is 
important to consider how these preferences might develop. In an 
unconstrained setting, oredicting the preferences or tastes of 
dec^ision makers js probj^atic. Wi^h the exception of some'global 
notions of hedonic pr^rences or the idea that, in general, people 
will choose to be "better off " rather than "less well off," it is vir- 
tually impossible to predict a priori what set of outcomes a given 
decision maker is likely to value. In organizations, however, the 
situation is far more predictable. Decision makers* exist in settings 
which act to constrain both the range of outcomes which might be 
preferred in situations where there is a choice and the preferences 
for particular outcomes within this reduced set. These contextual 
constraints are typically imposed through systems of goals, control 
systems, and incentives. Decision makers are usually constrained 
first by the nature of the job, that is, organizational or subunit goals 
are imposed along with the responsibility to work toward attaining 
these goals. To ensure that this is accomplished, some form of con- 
trol system is typically used which allows superiors in the hierarchy 
to monitor the achievement of subordinate decision makers. Upon 
discovery of a variance, sanctions can be applied. Thus, the orga- 
nizational context acts to constrain decision makers to pursue a Um- 
itedliumber of goals and to reward or punish them for success or 
failure. 

Without digressing into a lengthy discussion of goals and goal 
formulation processes (cf , Mohr, 1973, Perrow, 1970), it should 
^be noted that this process involves the translation of often vague. 
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non-operational, long-term goals into more specific, short-term 
outcomes. This activit> may, as has been empirically demonstrated 
(e.gi, Pfeffer & Salancik, 1978), be accompHshed through a polit- 
ical process of negotiation, bargaining, co-opting, coalition for- 
mation, and the garnering and application of power. The process 
may also result in the adoption of symbolic and non-operative goals 
whose purpose is often to satisfy external audiences and to provide 
for minimum constraints on organizational actors (e.g., Perrow, 
1970, Simon, 1964). Meyer and Rowan's (1977) interpretation of 
the symbolic nature of the operation of schools is an illustration of 
these goals. The concern In many public organizations with 
accountability is another goal, perhaps more relevant as a symbolic 
effort to reassure taxpayers, than as an operative goal relevant to 
effective and efficient organizational iunctioning. 

With respect to evaluation, Orlans (1973) has noted that ena- 
bling legislation is often written in vague terms in order to gain 
sufficient backing from legislators to insure passage. However, 
since program goals are not well-defined, it is difficult to develop 
measurable criteria, despite the fact that the legislation often man- 
dates an evaluation component. The lack of well-specified goals 
may encourage the political bargaining over <!:riteria described by 
Pfeffer and Salancik (1977). It may also result in evaluations which 
opponents claim are not appropriate. Program Head Start legis- 
lation, for example, offered no delineation of goal priority or con- 
crete objectives. Consequently, results of evaluation studies which 
showed no program impact were often ignored by decision makers 
as inappropriate or assessing only a part of the program's intent 
(Gordon & Morse, 1975). 

The result of the translation of organizational goals into short- 
term operative subunit goals, while perhaps an imprecise and 
ambiguous process, has several important ramifications for an 
understanding of the use of information by organizational decision 
makers. First, even when the goals are somewhat ambiguous, 
objectives act to focus the attention and efforts of those respon- 
sible. Second, although the goal may be imprecisely defined, con- 
trol systems are almost always developed which include measures 
of objective outcomes. ThesQ observable outcome measures act as 
highly salient foci against which performance may then be assessed. 
A large and persuasive body of research has documented the 
motivating effect of simple goal setting (e.g., Latham & Yukl, 
1975). For example, studies of budgeting (e.g., Wildavsky, 1974), 
public employees (Blau, 1964), students (Baum & Youngblood, 
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1975), homeowners (Erez, 1977), have all demonstrated that the 
nrtere setting of goals can act to direct behavior. Other research has 
. disc demonstrated that once an. individual becomes committed ^to 
an outcome through public actions, there is a tendency to continue 
to pursue the same goal, even if precec^ng efforts imet with failure 
(e.g., Salancik, 1977, Staw & Ross, 1978). These findings suggest 
that in an organizational context, decision makei^s vvill probably 
develop rather explicit preferences for outcomes. 

Finally, and most important, these preferences are solidified 
through the operation of organizational incentive systems, decision 
makers are rewarded for pursuing certain ends and punished for 
others. The pervasive impact of incentive systems should not be 
underestimated. Kerr (1975) provides a number of instances in 
which emplo>ees brehaved in seemingly contradictory ways, often 
appearing to do the opposite of espoused goals. For example, 
directors of orphanages were found to establish policies that worked 
against the placement of children in foster homes, and acted to 
keep them in the orphanage, universities routinely establish incen> 
tive systems that militate against encouraging high quality teaching, 
sports teams often reward individual performance when a team- 
oriented effort is requir^. Upoji analysis, the results werer^seen 
rtot to be contradictory but to be entirely consistent with the, 
incentive systems, that is, people in 9rganizatioris typically do 
those things for which they are rewarded. In Kerr's (1975) example,^ 
. for instance, orphanage directors who succeeded in placing chil- 
dren in foster homes would be "rewarded" by cuts in staff,' 
resources, and prestige among peers. In a direct test of the hypoth« 
esis that decision makers would respond to the operative control 
system, Harrell (1977) demonstrated that subordinate decision 
makers would follow their superior's lead and make decisions using 
similar criteria. This effect persisted even when superiors began 
making decisions which were contrary to the official policy. 

Obviously, the goals of survival and obtaining resources are 
probably more important to program administrators and staff than 
to independent evaluators. Iitfonnation which suggests that the 
program is not effective or efficient is likely to result in negative 
sanctions against those running t^e program. Clearly, there may 
be conflicts between the program administrator's goal of survival 
and the evaluator's goal of an accurate assessment. The incentives 
for those with vested interests in the program may be to devise 
strategies to^ minimize the collection, dissemination, and use of 
such unfavorable information, even if such actions go against some 
"objective** assessment of overall worth. 
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These findings are apphcable in understanding how decision 
makers develop preferences for outcomes, subunit goals are made 
salient, measurable, and rewarded, ^ontrol systems provide feed- 
back and sanctions which act to focus attention on achieving certain 
ends, failure to attain the desired outcomes ma> result in the loss 
of important organizational rewi^rds siich as promotion^ pa>, and 
status, as well as actual punishments such as demotion or termi- 
nation. As individuals in organization^ become more committed 
and less able to leave the organization, the importance of the out- 
comes is increased. Since there is seldom complete consensus 
among individuals and subunits on the. goals or outcomes .to be 
sought, the possibility of conflict and competition arises. Under 
these circumstances, it is possible that one person's gain is another's 
loss, hence, it is unlikely that decision makers will easily up 
preferences for certain outcomes if it rneans personal loss. 

Some indirect evidence of these effects may be seen in a review 
of 93 evaluation studies by Gordon- & Morse (1975). These authors 
found that evaluation researchers with some affiliation with the 
project being evaluated were far more likely to report the project 
as successful than \\ere non-affiliated evaluatqrs (52 percent to 14 
percent). Similarly, there was a greater tendency for non-affiliated 
researchers to report program failure than was the case among 
affiliated evaluators (32 percent to 14 percent). While not sug- 
gesting directly that affiliated evaluators deliberately biased their 
studies, the results are consistent with the hypothesis that incen- 
tives may lead to conscious or unconscious biasing of information. 

' The set of acceptable or preferred outcomes on which a decision 
maker is likely to focus is, in part, made salient through the spec- 
ification of a control system which assesses attainment of the 
assigned goals, incentive systems which sanction certain actions, 
and the committing effect of previous behaviors. These fiictors sug- 
gest the following proposition; 

Proposition 2/ Results of evaluation research are more likely {o be 
used by decision makers \vhen: 

a. the evaluation information can be used to assess achievement of 
quantifiable goals. 

b. the evaluation information is fed into a well-articulated and 
operating control system which incli^des an effective set of incen- 
tives; i.e., me*isured performance can be sanctioned. 

c. the evaluatioij information does not recommend actions which 
are incompatible with the existing control system's ability to 
monitor and sanction. 
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d. the evaluation information results frcgn studies performed by 
those who have a vested 'interest in the continuation of the 
project. 

INFORMATIONAL IMPACTS ON DECISION MAKING 

As suggested by Figure 1, information plays a central Yole in the 
decision-making process. Before information can have an impact, 
however, it must reach, and be processed by, the relevant decision 
makers. In an evaluation context this means that the evaluation 
report must not only be available to users but must also be read, 
.understood, believed, and acted upon. Failure by decision makers 
either to obtain the information or to process it can result in non- 
utilization. This suggests the need to examine tv^o separate steps 
irf the dissemination process: (1) how decision makers acquire 
information, and (2) how material is processed cognitively. A sub- 
stantial body of research is available which suggests the limitations 
and biases of organizational decison makers with respect to infor- 
mation acquisition and use. 

Information Acquisition 

Accessibility of Information. A large number of laboratory studies 
of information and decision making have documented the intui- 
tively reasonable conclusion that better quality information is gen- 
erally associated with imjft-oved decision-making performance (e.g., 
Porat & Haas, 1969, Streufert, 1973). Unfortunately, several authors 
have also noted that the majority of these studies may be overly 
structured when compared to the real-world situations they are 
meant to model (e.g., Connolly, 1977; Winkler & Murphy, 1973). 
Thus, while we have support for the importance of-fnformation in 
decision making, we need to examine the prdcess by which deci- 
sion Inkers acquire information before concluding that the iftere 
availability of better quality information insures improveJP per- 
formance. • . 

When we ekamine the evicjj^nce, doubts about the information- 
decision making link emerge.' Decision makers of many types 
appear to be noticeably biased fti their procurement of informa- 
tion. For example, in a direct test of the impact of accessibility and 
quality of information on information source use, O'Reilly (1979) 
found that although decision makers recognized information sources 
of high quality, they used sources which provided lower quality 
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information but were more accessible. O'Reilly (1979) explains 
these results in terms of the costs involved in obtaining informatipn 
from less accessible sources. Given that the decision makers were 
under time constraints and subject to numerous interruptions, it 
ma> have been that the> were simpl> unable to seek out higher 
qualit> information when it came from less accessible sources. Sim- 
ilar findings have been reported about the information-seeking 
behavior of physicians (Menzel & K^tz, 1955), scientists (Gerst- 
berger & Allen, 1968), policymakers (Clausen, 1973), and man- 
agers (Mintzberg, 1973). 

This bias toivards accessible information is also reflected in man- 
agers* strong preferences for oral as opposed to written information 
(e.g., Dewhirst, 1971) and for information from trustworthy or 
credible sources (e.g., Beach^jVlitchefl, Deaton, & Prothero, 1978, 
Giffin, 1967). Research in these areas has shown that managers 
t>picall> prefer shorter oral reports to longer written ones. Inter- 
estingl>, there is also evidence that when obtaining information in 
this manner, managers ma> judge the validity of the information, 
not on the facts of the matter, but on the credibilit> of the source. 
This nia> lead to the acceptance of a piece of information as ''true ' 
or "false" depending on hew much ihe recipient trusts the sender. 
Qlausen (1973), for instance, noted that Congressmen frequently 
cast votes on legislation based not on an understandmg ofjhe 
deeper issues but on the adv ice of a trusted colleague. This behavioi: 
is not necessarily bad, but simply reflects the inability of a Con- 
gressman to be fully informed on all issues. , 

The research oh soufqe credibility also suggests that it may be 
that it is,the 'safeness" or trustworthiness of the^ource, mor^ than 
expertise, which determines whether information is believed 
(0'Reill> & Roberts, 1976). The ramifications of this finding for 
underjyfanding the utiHzation of evaluation research ma> be impor- 
tant. 'Ev aluation units,- b> tfieir nature, may be perceived of as not 
sharing the same objectives as other subunits, i.e., not being trust- 
worthy in terms of source cxedibilit>. Tljese units may be seen as 
objective rather^than sympathetic, that is, expert but not triM- 
w»rth> . Thus, tnere is some possibility that the information pro- 
duced ma> be used less b> decision makers than information frgpi 
^"safer ' sources. David (1978), for example, fouad that the feedback 
of evaluation datJi which include^! personal explanations by eval- 
uation staff' was rnu<^h more likely to be utilized than a report that*^ 
vyaft merely delivered. ' - 
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Together, these biases may lead decision makers tdward pref- 
erences for information from particular, channels which they mAy 
characterize as accessible and trustworthy, and which provide con- 
densed treatments of complex issues. Information from these 
sources, as will be suggested later, may be concrete and easily 
assimilated, but not necessarily detailed or of the highest quality. 
\Tith respect to, the utilization of evaluation research, this suggests 
the following proposition: 

Proposition 3. Results of evaluation research are more likel> to be 
used by decision makers if they are: 

a. readily accessible to final decision makers 

b. summarized 

c. presented orally . - 

d. from a source deemed as credible i.e., trustworthy. 

Infonnation Sign. A" additional bias noted in studies of infor- 
mation acquisition b> decision makers is a tendency to avoid infor- 
mation which may suggest undesirable consequences. Janis and 
Mann (1977) offer several examples of this behavior describing how 
politicians and policy makers, when faced with unpleasant alter- 
natives, will avoid exploring ominous implications of desired courses 
of action^ Several examples of this sort are available in studies 
showing that program administrators have exhibited tendencies to 
dismiss negative findings (e.g., Borgatta, 1966; Carter, 1971). 
Goldstein et al. (1978, p. 33) report similar instances in which eval- 
uation results were selectively interpreted as offering support for 
a politically favored position. This willingness to avoid acquiring 
p^orative information and to seek out or selectively process favor- 
aple items has direct ramifications for evaluation research. As sug- 

^ jested earlier, decision makers often have vested interest* in 
pertain outcomes. This may predispose them to seek out infor- 
' niation which supports their position. David (1978) quotes a deci- 
sion maker as say ing, **I want infonnation to justify the expansion 
of the program" (p. 17). Bear and Hodun (1975), for example, 

^ found that subjects were likely to recall items of infonnation that 
were confirmatory to their position rather than those that were 
contradictory. Interestingly, this bias extended to recalling'' some 
contradictory items as supportive and even to recalling missing 
data as confirmatory for a preferred position. In an interesting lab- 
oratory study, Morlock (1967) demonstrated that it required sig- 
nificantly less infonnation for subjects to arrive at a decision 
favorable to their position than to arrive at ^decision considered 
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to be against their interests. In an organizational setting, groups 
of like-minded decision makers ma> exaggerate these biases toward 
selective perception and actuallv act collectivel> to censor or der- 
ogate information in opposition* to their desired ends. Janis (1972),^ 
labelling this process as "groupthink," prov ides a number of ret- 
rosi#ctive accounts in which groups acted to bolster desired opin- 
ion.s^and exclude contrar) ones. For instance, decision making b> 
PVesident Kenned>*s advisor) committee during the Ba> of Pigs 
incident was characterized afterwards b> the suppression of doubts, 
creation of feelings of_mianimit> and invulnerability, and an 
unwillingness to risk conflict within the gioup. Johnson (1974), in 
a studv of group decision making, prov ides a nice example of this 
tendency to seek^animit) and avoid conflict. She h>pothesized 
that executives would make a less desirable, but acceptable, short- 
run decision to avoid generating conflict with others in the group. 
Using "^businessmen across ten situations she discovered that 
althougfsubjects could ^entif> the ideal decision for each situa- 
tion, their overwhelming tendency was to make a sub-optimal 
decision in order to avoid conflict. David (1978) again quotes a user 
of ev aluation data as saying, "I look at test scores mainl> to confirm 
my own impression. If they differ, my impression counts * (p. 16). 

These biases, that is, tendencies to avoid information which sug- 
gests undesirable consequences, to seek out supportive informa- 
tion, to require less supportive information to arrive at a favored 
decision, and a desire to avoid making decisions which will gen- 
erate conflict) suggest the following proposition: 

Proposition 4. Results of evaluation research are more likely to be 
used by decision makers if the information: 

a. isvsupportive of the outcomes favored by the decision makers. 

b. does not l^adjo conflict among the set of relevant actors. 

Proposition 4 appears most appropriate when, as Jftnis and 
Mann (1977) suggest, the group is cohesive and members value 
their as.sociations. Under these circumstances, it may be that group 
members will be more apt to suppress conflict in the name of group 
unanimit). When evaluation information is available to competing 
factic^s or decision makers ^)elonging to differing groups, the fol- 
lowing proposition is suggested: 

Proposition 5. Results of evaluation research are more likely to be 
sought out and used hy decision makers under competing or con- 
flicting conditions when: 
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a. Infonnation supportive of an undesired position is availAle to 
actors seeking different outcomes. 

b. The decision-making process is drawn out over a long period of 
time which allows alternative positions to be presented. 

Communication of Information. A final set of potential biases on 
mformation acquisition b> decision makers in organizational set- 
tings is related to the communication of infonnation in organiza- 
tions. Several authors have noted that communications in 

.organizations may be withheld or distorted (e.g., O'Reilly & Pondy, 
1979, Wilensky, 1967). Subordinates have been found to be biased 
toward passing superiors information which reflects favorably on 
the subordinate and suppressing unfavorable information. O'Reilly 
(1977) discovered that subordinates who did not trust their supe- 
rior were willing to suppress unfavorable information even if they 
knew that such infonnation was useful to their boss. Other inves- 
tigations have shown similar results. For instance, Pettigrew(1972) 

. documented how a single individual, acting as a gatekeeper for 
information flowing to a policymaking group, was able to deter- 
mine the outcome of a purchasing decision by carefully allowing 
only certain types of infonnation through to the decision makers. 
Plott and Levine (1978) demonstrated how, through the arrange- 
ment of the agenda of a meeting, outcomes could be determined 
in advance. Lowe and Shaw (1968) provided evidence that depart- 
ments systematically inflate and bias budget requests to support 
claims for increased resources. In a slightly different vein, Kaufinan 
(1973) showed how subordinates learned not to pass certain items 
of infonnation upward in tfie hierarchy because superiors, upon 
learning of these, would be required to act in ways contrary to the 
subordinate s self-interest. A classic example ofthis system-induced 
distortion is provided by McCleary (197*^ in a study of how parole 
officers report clients' violations, observing that parole officers 
under-reported deviant behavior to their supervisors. As noted by 
Kaufman (1973), subordinates are often punished for providing 
accurate reports. In McCleary's study, this resulted in incidents 
being reported only when the infonnation sent upward would 
enhance the subordinate s career. 

Numerous other examples are available which attest to the fact 
that information is often selectively filtered and distorted as it is 
communicated in organizations (e.g., Allison, 1971; Janowitz & 
Delany, 1957, etc.). Such filtering and distortion appears to result, 
most often, from individuals or groups attempting either to gain 
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desired outcomes ^such as increased resources or power) or to 
avoid their loss. When considered in conjunction with th^ biases 
toward reliance bv decision makers on short, oral reports from 
trusted sources, the impact of distorted information ma> be 
heightened. 

Cleady, these biases have important consequences for the trans- 
mission and use of evaluation research since interpretation of the 
results of evaluation studies mav be biased either to support or 
refute a particular position. If, as NIarch and Sirrton (1958, p. 165) ^ 
note, "inferences are drawn from a bod> of evidence and the infer- 
ences, instead of the evidence itself, are then ct^mmunicated," 6he 
opportunity for subtle distortion is magnified. Findings which have 
undesirable consequences ma> be withheld b> superior gate- 
keepers also. Direct e\ idence of this effect is recounted b> Coleman 
, (1972) who describes how HEW attempted to minimize the impact 
of an EEO stud> because its findings were inimical to the interests 
of some othm HEW agencies. Here we find decisions being made 
to-suppress or alter information both to minimize conflict and to 
avoid undesirable outcomes. 

It should be noted that not all distortion in organisational, com- 
munication is necessarily intentional. With an> transmission from 
one individual to another there is almost always some degradation 
of the message (e.g., see Campbell, 1958). Receivers of informa- 
tion recall certain parts of messages and forget or minimize othep^, — - — 
This unintentional distortion, due to differences in cognitive tuning, 
ma> be increased when the communication occurs between groups 
» who use different vocabularies, are sensitive to different goals and 
constituencies, or are using different criteria for determining what 
is important. Janis (1972), for example, describes how information 
during the attack on Pearl Harbor was not transmitted because 
senders, unaware of the broader picture, did not perceive certain 
information as important. Evaluation researchers, whqse concern^ 
are often somewhat different than administrators,' ma> not full> 
appreciate the concerns of users of their studies. The use of a dif- 
ferent v(>caf>ular> and the requirements for communication through 
several hierarchical levels may lead to evaluation reports which 
administrators see as not timel>, too technical, focused on issues 
which are not central, and generally unconvincing (e.g., Alkin, 
1975, Cox, 1977). Such opinions on the part of administrators, cou- 
pled with previously mentioned biases such as reliance on acces- 
sible and credible information sources, make it clear that evaluation 
information mav not have the impact expected b> e^valuation 
researchers. ^ 
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Acknowledging that information may be blocked or altered 
during communication within organizations suggests the following 
- proposition: 

Proposition 6. Results of evaluation research are less likely to be 
useful when: 

a. transmitted through several intermediate links in a communi- 
cation network rather than delivered directly to the relevant 
decision makers. 

b. transmitters of the information are likely to suffer personal or 
, organizational losses from the message. 

c. senders and receivers do not trust each other. 

d. the information is transmitted to decision makers in another 
agency or organization. 

e. the information is transmitted to decision makers in a different 
functional group within the same organization. 

* ' ♦ 
Information Processing ' 

Discussion to this point has focused on biases which may afiect 
^ the acquisition of information b> decision makers in an organiza- 
tional context. Once the information has been acquired, however, 
it may still be that individuals will fail to process it accurately. A 
diverse and well-developed bod> of research is available which 
documents these limitations in cognitive information processing 
(e.g., Slovic, Fi^hhofF, & Lichtenstein, 1977). Two general themes 
from this research are relevant to an understanding of the use of 
evaluation research by decision"" makers. (1) factors which limit 
one's ability to assimilate information, and (2) processing strategies 
which may result in inaccurate or misleading inferences. 

Cognitive Limits on Information Processing. Early studies of 
human information processing demonstrated unambiguously that 
decision makers could only use limited aiYiounts of information. 
xMiller (1956), in a classic study, demonstrated experimentally that 
short term memory is limited in most people to seven "chunks** 
of information plus or minus two (a "chunk" being the largest single 
item of information recognized by the processor), and that an 
individual's ability to "chunk*^* information acts as a direct con- 
straint on the input of data. What is important to us is that this 
fundanrjental physiological limitation appears to^act as a constraint 
on all decision fhakers. Numerous studies have demonstrated t)iat 
physicians, stockbrokers, meteorologists, policymakers, and a 
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variety of other "experts" all appear to use only a very limited 
number of cues in makmg complex decisions (e.g,, Dawes & Cor- 
rigan, 1974, Slovicet al.rT977, Slovic & Lichtenstein, 1971). Other 
research in this area has also shown that variables such as person- 
ality, , cognitive structure, and demographics are related to the 
ability to process information (e.g., Nystedt, 1972, Schroder, 
Driver, & Streufert, 1967, Taylor & Dunnette, 1974). Further evi- 
dence of limitations on the ability to process information is avail- 
able in studies showing the biased nature of memory and 
perception. Buckhout (1974), for example, lists three common 
sources of unreliability in recall. (1) insignificance in the original 
situation, i.e., cues which were later deemed important were not 
closely attended to at the time, (2) the degradation of information 
in menjory over time, and (3) pressures and distractions on the 
information processor which reduce the amount and accuracy of 
information used. 

Each of these limits is potentially important for an under- 
standing of how decision makers may use evaluation information. 
For instance, given that decision makers are able to use relatively 
little information, it becomes problematic to know which infor- 
mation it^ms of the total quantity available a decision maker will 
focus on and use. Studies of experts are consistent in finding that 
experts on the same subject ty pically use different information in 
making expert judgments (e.g., Slovicet al., 1977). This suggests 
that decision makers, when presented with evaluation research 
results, may interpret and weight the information differentially. 
Hawkins et al. (1978), for example, in a drug evaluation Jtudy^ 
showed that various actors were weighting information differently 
according to the evaluation criteria they were using. Other studies 
have also demonstrated variations in preferences for types of infor- 
mation across decision makers (e.g.^ Kilmann & MitrofF, 1976), as 
well as in how stress reduces one's ability to process information 
(e.g., Wright, 1974). Since, as Mintzberg (1973) has shown, man>> 
agers' work is charactetistically fragmented and subject to distrac- 
tions and time pressures, it is likely that users of evaluation 
research will be unable to assimilate fully all the information con* 
tained in a report. Instead, users are likely to form coverall impres- 
sions, subject to the biases mentioned previously, and to weight 
the results accordingly. This interpretation is consistent with studies 
which show that with the passage of time details are forgotten, that 
the reconstructed meaning is often less ambiguous than originally 
portniyed, and is then interpreted as offering support for a favored 
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position (e.g., Buckhout, 1974, Ross, 1977), that is, information 
contained in the original signal which is unfavorable is likely to be 
either forgotten or reinterpreted so as to minimize its negative 
consequenGes. 

These and other studies of individual limits on cognitive infor- 
mation processing corroborate ^ reasonable but often neglected 
fact, memory is a selective and often falfible source of information. 
The perceptual process of detecting and attending to stimuli is 
affected b> factors such as stress. With the passage of time there 
is also a tendency to reorganize our **memory" into coherent rec- 
ollections b> stripping awa> contradictor) evidence and filling in 
an> gaps with * constructed*' facts. These limitations suggest the 
following propositions: 

Proposition 7. Only limited amounts of information from an evalu- 
ation study are likely to be used for decision-making purposes. 

Proposition 8. Given the same evaluation information, different 
. experts will use different parts of the report in different ways, that 
is, judges will weight differently the same evaluation information. 

Proposition 9. Over time, decision makers will be more likely to 
interpret favorable information from an evaluation study^as less 
' ambiguous than originally perceived. 

Proposition 10, Over time, decision makers will be more likely to 
forget unfavorable information from an evaluation study or reinter- 
pret such information as either irrelevarit (e.g., does not address 
the *Velevant" question) or favorable. 

Selective Processing. Aside from these limits on decision makers* 
ability to process information, there also exist biases in the manner 
in which information is cognit^ely processed by individuals. Three 
of these biases have direct ramifications for evaluation research 
utilization. (1) selective perception, (2) self-serving biases, and (3) 
a preference for vivid, concrete information for use in decision 
making. The first relates to the tendency, described in the pre- 
vious section, for users to "reinterpret** acquired information to fit 
preconceptions or to allow the user to maintain a consistent. set of 
attitudes and behefs about a given topic. This bias toward consis- 
tency injected by the reconstruction of facts is dramatically 
increased through the three selective processing mechanisms men- 
tioned above. 
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Janis and Mann (1977) offer a number of excellent illustrations 
of instances in which decision makers either defensively avoided 
acquiring or processing unfavorable information or bolstered their 
position through the selective acquisition and interpretation of 
favorable data. This tendency may be seen in decision makers* 
willingness to overweight negative information when they desire 
to make a negative decision (Kanouse & Hansen, 1972). In these 
circumstances, decision makers wishing to reject an opposing view 
have been shown to use whatever negative information is available. 
Selection interviewers, when presented with a large number of 
positive cues and ver> few negative ones, have been shown to 
attend to the negative information systematically and to use it to 
reject applicants even though the positive information is far more 
potent. Miller and Rawe (1967), for example, found that when sub- 
jects were required to make assessment decisions, there was a sig- 
nificant tendency to be influenced by negative rather than positive 
adjectives used to describe a candidate. Other corroborative evi- 
dence is available from studies of personal perception among those 
who make investment decisions, among gamblers, and others 
(Kanouse & Hansen, 1972). Whejn^ecision makers favor a posi- 
tion, the bias has been shown^o operate toward tbe selection of 
favorable information as well (e.g., Morlock,1967). 

It should be noted that this bias does not necessarily suggest 
that decision makers truncate their search for information once 
having obtained data which can be used to support a desired 
position or oppose an undesired one. The apparent tendency is to 
selectively seek out information which bolsters one's position and 
avoid unsupportive information in either acquisition or processing, 
, but not necessarily to avoid searching. In fact, a number of labo- 
ratory studies have demonstrated an interesting propensity among 
decision make's to desire more information than can be effectively 
used (Chervany & Dickson, 1974). The paradox is that decision 
makers appear to seek more information than required, even to 
the point of inducing overload. While the overload may actually 
impair performance, the additional information has been shown to 
increase the decision maker s confidence (Chervany & Dickson, 
1974, Oskamp, 1965). The net results may be that decision makers 
arrive at poorer decisions but are more confident in their choices. 

Thus, it may be that decision makers w\l\ selectively seek out 
information which supports or opposes a position, acquire as much 
of this information as possible, and be increasingly confident in 
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their decision although such decisions may be substantial!) biased. 
Meltsner (1976), in a book on policy analysts in bureaucratic set- 
tings, makes a relevant distinction between two categories of infor- 
mation sought by decision makers, that is, information used to 
Tuake decisions and information used to support decisions. The 
latter categor> is indicative of the type sought by decision makers 
to justify a position. Meltsner describes at some length how it is 
not uncommon for decision makers to hire outside consulting 
groups to do evaluation studies, not to be used for decision-maldng 
purposes but solel> to prov ide credible information which supports 
a decision which has already been made. This idea is directly 
related to the earlier discussion of information as a political resource 
and suggests the following proposition: 

Proposition 11. Results of evaluation research are more likely to be 
used if portions of the study can be selectively interpreted as either 
supportive of a desired position or unsupportive of opposing 
positions. 

i% 

^ Self-Serving Biases. Aside from propensities to perceive and 
process information selectively, decision makers have also been 
shown to engage consistently in what has been labelled "self- 
serving* biases, that is, researchers have noted that individuals 
often view themselves more favorably than seems objectively war- 
ranted (e.g.. Miller & Ross^lQTS)^ Thus, for example, bettors have 
been shown to overpredict their gambling successes consistently 
(Blasovich, Ginsburg, & Howe, li975), production managers to 
overpredict their performance (Kidd & Morgan, 1969), and cor- 
porate presidents to overpredict their firm s success in meeting 
competition (Larwood & Whitaker, 1977). Aside from this future- 
oriented optimism, investigators have also shown that, in retro- 
spect, members of successful groups see themselves as more 
responsible for their groups than do members of groups that have 
failed. Schlenker ancl Miller (1977), for example, found that mem- ^ 
bers of groups that had failed assigned less responsibility for the 
group's poor performance to themselves than they typically assigned 
to- any other member of the group. 

In organizations, such pervasive "self-serving" biases have 
important ramifications for the acquisition and Interpretation of 
information. These biases become especially important when par- 
ticipants are linked to, or are responsible for, previous decisions 
to allocate resources. Under these conditions, decision makers may 
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be committed to a particular program. Staw and his colleagues 
(e.g., Staw, 1974, Staw & Fox, 1977) have demonstrated how com- 
mitment to d course of action ma> result in escalating commitments 
of resources fo failing projects. Staw and Ross (1978), for instance, 
demonstrated that polic>makers who had allocated resources to 
projects which subsequently failed for reasons that they should 
have foreseen, were more likely to devote even more resources to 
the project in succeeding time periods than were decision makers 
who had sponsored successful projects or whose projects had failed 
for exogenous reasons beyond the decision maker^s control. This 
example is similar to foreign policy failures such as the United 
States' involvement in Vietnam. Staw and Ross quote George Ball 
(1965), who in the early years of the Vietnam War stated that, 
"Once we suffer large casualties . . . our involvement will be so 
great that we cannot — without national humiliation — stop short of 
achieving our complete objectives." These examples are disturb- 
ingly similar to certain Educational programs which persist in spite 
of evaluation research which documents their failure to achieve 
stated objectives. 

Two possible reasons for such events can be offered. First, the 
stated objectives of the program which are evaluated and found 
wanting are not representativ e of either the true objectives of the 
program (for example, when a program is established for political 
purposes and evaluated on educational attainments), or the eval- 
natron is^ done- on only a sirbset of the total program goals. Or, 
second, evaluation information attesting to the program's inade- 
quacies is ignored. It is in this latter instance that the effects of 
commitment described by Staw may be lixiked uith self-serving 
biases. 

It may be that the ability of individuals to be over-optimistic 
about future events predisposes them to commit themselves to 
courses of action. Further, when cohesive groups are involved, 
there may be an even greater impels to choose risky options. 
Once committed, the selective perception biases described earlier 
can act to provide information supportive of the original decision. 
Halberstam (1972) provides numerous illustrations of how Robert 
McNamara and others engaged in this activity during the war in^ 
Vietnam. Gouran (1976) provides similar examples showing that 
Nixon and his aides persistently discounted the importance of evi- 
dence during the Waterj^ate cover-up. Selective perception biases 
allow the parties involved to choose information, as suggested in 
Proposition 11, which either supports the aims of the programs or 

-50 • 



52 



Charles- O'Reilly III 



rebuts opposition claims. Self-serving biases may also allow for the 
development of a false consensus or the illusion that one's behav- 
iors and choices are common and appropriate while opposing 
responses are uncommon and not widely supported. These self- 
serving biases also act to make the decision maker reluctant to 
abandon a chosen course of action. As long as information is avail- 
able which can be interpreted as supportive of a given position, ^ 
the bias on the part of the central actor will be to focus on this 
corroborative information. As PfefFer and Salancik (1977) have 
showTi, when ambiguit> exists, particularistic criteria can be used 
b> decision makers, that is, unless a widespread consensus exists, 
it is possible for opposing decision makers to argue for their posi- 
tions and to selectively use information to support their claims. 
Given that evaluation is often prescribed precisely because the 
situation is ambiguous with respect to a given program, it is 
obviously the case that the selective processing oY information will 
occur. Under these circumstances, individuals are likely to be 
involved and committed to particular points of view and self- 
serving biases will be operating. This impact may be heightened 
since the results of evaluation research ma> also be ambiguous. 

A final effect of self-serving biases may be seen should a program 
be declared a failure, either for political or substantive reasons. 
Under these circumstances, self-serving biases may allow involved 
participants to cognitivel> reconstruct their involvement and 
devalue their responMbility for the failure, selective perception 
ma> act to focus on exogenous ev ents which explain lack of success 
in terms of others' actions and unforeseeable events. Thus, the 
operation of self-serving biases suggests the following propositions 
relevant to an understanding of the use of evaluation information. 

Proposition 12. Responsibility for a program resolts in increased 
commitment pn the part of decision makers and increases the like- 
lihood that the results of evaluation research will be used when. 

a. the evaluation information can \>e interpreted as favorable or 
supportive of the program. 

b. the evaluation information is ambiguous and can be argued as 
not reflective of the overall scope of the program. 

c. no strong consensus exists as to the specific goals of the program 
such that some evaluation information may be interpreted as 
favorable to some parts of the program. 

Proposition 13. Results of evaluation which are unfavorable to a pro- 
gram are more likely to be used when: 
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a. there exists a set of actors with objectives contrary to those 
associated with the evaluated* program. 

b. resources are scarce, so that the competition for resources is 
greater. 

Abstract Versus Concrete Information. Typically evaluations are 
* conducted to assess whether a program meets its goals, is suc- 
cessful compared to other projects, or in order to provide for the 
feedback of information to improve performance in succeeding 
time periods. Underlying these reasons is the idea that evaluation 
information will allow decision makers to derive inferences about 
causal relationships (for example, between educational interven- 
tions and student achievement). An important postulate of this 
process holds that causal explanations will be influenced by con- 
sensus information, i.e., informaticJn concerning base rates and 
how a given project fares compared to the base rate. Attribution 
tfiWy researchers (e.g., Nisbett, Borgida, Crandall, & Reed, 
1976) have drawn attention to the substantial amdunt of evidence 
which has failed to support the* postulated effect of consensus 
.j^ information. For instance, Tversky and Kahneman (1974) have 
, | demonstrated this point b> asking subjects to judge the probabilit>' 
that a target individual, described in a brief personality sketch, was 
an engineer, given; 

\ 

\ (a) that he was drawn from a population of 70 engineers and 30 
j irtawyers, or 

( (b) that he was drawn from a population of 70 lawyers and 30 
* engineers. 

Knowledge of the population base rate for occupational categories' 
had no effect whatever on judgments of the prpbability that the 
target individual was an engineer. Instead, subjects relied exclu- 
sively on the personality sketch and based the decision on the 
degree to which the description fitted the stereotypic engineer or 
lawyer. Numerous other examples are available which demon- 
strate that decision makers, and even scientists familiar with sta- 
tistics, habitually ignore information about the popul^[tion and 
draw recklessly strong inferences about the underlying population 
from knowledge of a ver> small sample. Tversky and Kahneman 
(1974) refer to this as the *'law of small numbers/* 

Observers offer some reasons why even expert decision makers 
Ignore base rate or consensus information. Kahneman and Tversk>', 
. for example, speculate that people may not know how to combine 
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base r^ite information and, therefore, ignore it. Nisbett, et al. 
(1976) propose that base rate information, b> its ver>' nature, is 
abstract and pallid, and ma> simpl> lack the force to persuade sub^ 
jects to attend to it and to use it. People, it seems, are unmoved 
b> dr>, statistical data dear to the hearts of scientists ^nd evalua- 
tors. As fiertrand Russell observed, "Popular induction depends 
on th^ emotional ijiterest^of the instances, not upon their numt^.'* 
Individuals respond to vivid, 'concrete information and ignore 
abstract data. 

Nisbett, et al. (1976) offer several examples of this tendency. 
Consumers have long ignored medical advice to quit smoking and 
safejty advice to fasten. seat belts. Such appeals typic>all> report 
numbers such ius the probabilit> of being a v ictim and are hirgel> 
ignored. Yet when a highly visible and concrete incident occurs 
vrhich people can focus on, results often dramatic. For example, 
medical exhortati^ on the value of early detection of breast 
tumors long went virtually unheeded. Yet, the waiting lists at 
cancer detection clinics became months-long after television 
reported the mastectomies of Mrs. Rockefeller and Mrs. Ford. In 
an interesting experiment, Borgida and Nisbett (1977) provided 
prospective students with course evaluations based on ratings of 
students who had previously tiiken the courses. This information 
had little impact on course choices. In. contrast, brief face-to-flice 
comments al>out the courses had a substantial impact on course 
choices. Other studies have noted how^ vivid information which is 
non-diagnostic, that is unrelated to the decision to be madp, may 
have an impact on the choice (Troutman & Shanteau, 1977). 

T^onsider the ranjifications of these biases for the use of evalu- 
ation information. The typical evaluation report is a document 
which relies heavily on the statistical analysis of data and variations 
from mean levels of performance. The essence of such a report is 
on base rates and variations from the mean. Complaints by con- 
jkimers of such reports that the information is "not helpful," is "too 
dry," 'Velies too much on statistical analysis," or "doesn't get at 
t^ie real problem" may, in fact be symptomatic of decision makers' 
mability to use abstract information. On the other hand, users of 
dvaluation information often focus on a single, concrete, often dra- 
matic, example even though the chosen example may not be rep- 
resentative of the larger picture. Patton (1978), in his discussion 
of the meanings of evaluation data, calls for evaluation reports 
which represent something meaningful to the identified informa- 
tion users, including efforts to reduce the mysticism of scientific 
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jargon and to increase the iace^lidit> of measures. Clearly, con- 
crete^nd vivid examples which are accural representations of the 
underlying results are likely to be a(?cepted and rememberecfoy 
deciiJion makers more easily than compilations of statistics. Meltsner 
(1976, p. 234), reports the advice of a chief federal analy st that 
when writing two-page summaries of reporte, it is important to 
**sprinkle them with juicy^punch lines that will catch the i/eaders' 
interest. " Such pragmatic adv ice reflects the fact that not ^nl^^j^ 
decision makers busy enough not to want to read reports, but they 
are also more likely to remember, and 'therefore more likely to 
use, vivid information. Another analyst reported that half his time 
was spent as a "rewrite man" trying to translate statistical material 
inty a form which would be meaningful to i-he President and White 
House staff and agency heads. These observations lAadersaore the 
bias people have toward concrete information and suggest the fol- 
lowing proposition: 

^Proposition 14. Results of evaluation research are more likely to be 
used when vivid, concrete illustrations of the conclusions are 
available. " ^ ' . 



SUMMARY AND CONCLUSIONS 

^ This ti^latment of organizational decision makers' utilization or 
non-utilization of evaluation information began with xi simplified 
model of decision making whose purpose was to highlight the 
importance and potential in34)acts of information in the decision 
process. I then argued that, in-organizational settings, rather than 
following a decision-making process which proceeds from*n problem 
to alternatives to a^hoice whrtHi optimizes, organizational decision 
makers have strong preferences for certain outcomes and act, in 
their information search and processing, in way s^ calculated to max- 
imize the attainmei^t of desired eqds. This view recognizes that 
information is only one cajrimodity which may help or hinder goal 
attainment. This' view also recognizes the fact that sorr^ choices, i> 
arc unac^^eptable to certain decision makers, regardless of the net / 
benefit to the i^ger collective. Knowledge does not necessaril/^ 

^equate with action. 

Given the political process through which -goals and objectives 
are negotiated among groups of organizationai participants (e.g., 
Pfeffer & Salanoik, 1978), universal agreement on any allocation 
of res*0n>jces is unlikely. This lack of consensus makes the^ process 
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of organizational decision making a political one, often character- 
ized by conflict and disagreement. In order to achieve a semblance 
of rationality, if onl> to satisfy constituencies outside the organi- 
zation, I argued that procedures were established which, though 
they give the appearance of comprehensive rationality% may be 
more symbolic than real* Within the bounds of this ^'procedural 
rationality** I argued that individual^decision makers were **con- 
textually rational," that is, attempted to m^imize goal attainment 
given a set of^ituational, organizatiojjal, and individual constraints. 
Thus, it may be that the requirement for evaluation is a manifes- 
tation of the need for procedural rationalit> in public management 
whilenhe actual use of evaluation information is subject to the con- 
textual rationality of relevant decision makers. That some evalua- 
tions are conducted as pro forma arrangements with funding 
agencies to insure compliance v^ith regulations, but not intended 
for use in decision making, can be interpreted as an example of 
this procedural rationality. The fact that other evaluations are con- 
ducted to support previously made decisions may be an example 
of the contextual rationality of decision makers. 

Since the decision-making process as illustrated hi Figure 1 i$ 
an interactive one, and since the argument is that much of fh^ 
information manipulation stems from the preferred set of pyt- 
comes, it is important to consider how these preferences are 
developed as well as how information is us^d. I proposed tha,t two 
pnmary sets of constraints were relevant. First, in organizational 
settings, decision makers are seldom indifferent about outcomes. 
Rather, goals are assigned, for example, profit or cost margins, and 
control systems established to monitor and sanction responsible 
mdividuals. Power, or the ability to induce other groups or indi- 
viduals to behave in prescribed ways, becomes an im^ortant^ con- 
sideration for goal attainment. The effect of these^^^anables on 
decision makers is to make both salient and desirable a limited set 
of outcomes. These constraints, when cpupled v^dt^ potential loss 
of personal rewards such as status, promotion, soci^^approval, and 
money, act to commit decision makers to certain outcomes. 

I argued that decision makers, once committed, were then 
potentially subjectto biases in both the acquisition and processing 
of information for use in decision making. Evidence' was cited 
which demonstrated that decision makers were biased in their 
search, for information, preferrir!| accessible info^tnation which ^ 
supported their preferences rather than contrary information, even 
if such information was of higher quality. Fu|-ther, evidence was 
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also available which documented how commitment to certain 
desired outcomes was associated with the distortion of information 
in organizational ct)mmunication. Hence, it ma> also be that the 
information available to unbiased decision makers may, if it hais 
been transmitted through an organizational hierarchy, already con- 
tain inaccuracies or distortions. 

The information processing of decision makers was also consid- 
ered as a potential source • of non-utilization of evaluation infor- 
mation. In pursuing desired outcomes, decision makers were often 
shown to perceive and interpret information selectively. It has Also 
been shown that human information processors do not deal well 
with dry, statistical data, but prefer more vivid, concrete exam- 
ples, even though such information may be inaccurate or mis- 
leading. These biases may be important since evaluation information 
is typically quantitative and statistical. It was argued thaf the com- 
bination of decision makers* selecnve perceptions of supportiv e 
information and general preference for vivid examples biases them 
away from the use of evaluation information unless such informa- 
tion is .supportive. 

The joint effects of the situational and individual constraints on 
information use by decision makers is outhned in Figure 2. Con- 
text variables such as incentive systems, group norms, and orga- 
nizational structure may act to affect the information which is 
available to a decision maker. Context variables, as well as the 
manner in which information isjprocessed cognitively, may also act 
to affect individual preferences for certain t^pes of information. In 
turn, these variables may determine how^ and what information is 
used by decision makers. 

When evaluation research is considered from the perspective 
developed in this chapter, and subject to the constraints presented 
in Figure 2, several observations about utilization of evaluation 
informatiw^e noteworthy . First, evaluation research, regardless 
of the rtger vhth which the study was conduotSil, is not likely to 
be regarded by decision makers as objective, nonpartisan infor- 
mation. Rather, such information will likely be viewed as useful 
to some interested parties, threatening to some, and irrelevant to 
others. The utilization of such information in decision making will 
probably reflect not any objective measure of quality of the research, 
but a number of factors ind^epetident of the evali^tion study such 
as the degree of consensus or conflict among those involved in the 
decision process, the relative power of the participants, pressures 
on the primary decision makers, availability of other information, 
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etc. In some cases, individual characteristics of the decision maker 
mav also determine the utilizatioa^non-utilization of evaluation 
results. When studies are undertaken, not to provide information 
for decisions, but as a means of increctsing confidence in a position 
or for s>mbolic reasons, it is unlikely that an> direct impact of the 
research will be observable. This does not impl> that such infor- 
mation IS not useful, onlv that its function is not directl> related 
to decision making. 

Finallv, the propositions developed in this chapter are clearly 
tentative and somewhat simplistic and onl> suggest possible 
hypotheses which might be tested empirically. Obviously, there 
are a large number of influences on the use of evaluation research 
which have been omitted here. Therefore, what is proposed in this 
chapter is not a well-articulated theory of utilization of evaluation 

decision making, but some tentative propositions based on pre* 
vunis research on organizational decision making and information 
use. 

Future Research 



Based on the empirical and theoretical evidence reviewed here, 
several tentative recommendations appear warranted. First, as 
suggested by David (1978), efforts to improve the quality of data 
are not likely to affect local use of evaluation. Instead, local ev al- 
uators might, as suggested in Figure 2, be attentive to under- 
standing the goals and constraints operating on local decision 
makers. Rather than presuming that evaluation data are neutral 
and decision makers are rational, attempts might be made to pre- 
sent data in ways in which decision makers are likely to feel least 
threatened. These might include strategies to reduce commit- 
ments to competing goals, to increa.se trust by receivers, and to 
target data to specific goals. 



.\Gtions such as these may improve the utilization of evaluation 
information by decision makers. One may question, however, 
whether this effort is desirable. As David (1978) discovered, one 
of the primary functions of local evaluation Units is to meet external 
reporting requirements, not to serve as a guide for program 
improvement. Other sources of information may be more useful 
in improving local progiariis. Perhaps local evaluation units should 
be serving more as a data collection and evaluation center for var- 
ious interest groups as well as being involved with assessing stu- 
dent achievement, i.e., more responsive in terms of the provision 
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of information than with evaluation per se. Certainly the evidence 
reviewed here would suggest ihat there are maay "good'' reasons 
wh> interested decision makers might not accept information from 
a local evaluation agenc>. If these evaluation units are to have an 
effect on decision making and program revision, more attention 
needs to be paid to t'he interested parties' constraints and biases. 
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Notes on Zucker's "Role of 
Evaluation Units in Schools' 



Introduction 

Lynne Zucker is a sociologist at UCLA. In her chapter, she uses 
data from the CSE survey of school district evaluation units to 
examine the. role that these units play in signaling to external 
funding sources that their parent district organization is complying 
with regulations and is therefore entitled to continued support. 
Before re-analyzing the CSE survey data, Zucker introduces sev- 
eral propositions about how formal structures develop in organi- 
zations such as school districts. She contrasts the emergence of 
these structures in school districts, which must be responsible to 
societal demands, with the emergence of formal structures in other 
institutions which are more technically or task-oriented. Her 
chapter builds on the conceptualizations of Weick (1976) and 
Meyer and Rowan (1977) who have described how the technical 
core activities of educational organizations are disconnected from 
or only loosely coupled with formal administrative control 
stnictures. 

Argument 

Zucker posits, in agreement with the authors cited above, that 
schools and school districts belong to a class of organizations 
regarded as essential in achieving societal goals. Because of this, 
man> of their inrernal operating units are initiated and maintained 
primarily to respond to External expectations or requirements. 
While evaluation units ma> sometimes serve the internal operating 
needs of the organization, their work is frequently irrelevant or 
tangential to the organization's technical tasks. 

Zucker notes that societal demands on school districts, in the 
form of federal and state laws and regulations, have increased in 
recent >ears partly because of the growth of federal- and state 
fihancial support to local educational agencies. Such financial sup- 
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port has been distributed in accordance with categorical program 
guidehnes and continued in hght of evidence about the program's 
effectiveness. 

To produce that evidence, man> school districts have organized 
separate units to do evaluation and testing. Zucker contends that 
these evaluation units perform the signaling function alluded to 
earlier, that is, the> signal to groups or agencies in the environ- 
ment that the district is in fact complying with legislative or 
administrative requirements. 

Zucker defends both the legitimac> of the demands made by 
federal and state agencies on school districts and the rationalit> of 
school districts in de-coupling the evaluation units, which respond 
to those requirements, from other sub-units within the district. 
She points out that evaluation units often collect data to tell the 
outside world thaf the district is performing adequately. Such 
adequate performance is the justification for additional resources 
to the district. 

This thesis has a number of implications for how personnel in 
evaluation units ma> regard instruction and for how teachers and 
administrators responsible for instruction may regard evaluation 
units. Since evaluation units are structures which fulfill the dis- 
trict's need to be externally accountable their staff tend to have 
minimal mfluence over internal organizational policies. They tend 
to emphasize to themselves and to others the ambiguity^ and 
uncertainty inherent in the technical — that is, instructional — 
activities of the district. As they self-protectively, and perhaps 
accuratelv, assert their own lack of control and their own uncer- 
taint> about causality in the m^tructional process, they tend to 
produce information of little internal use. Therefore, the status and 
power of the unit within the central ofiice tends to be minimal. 

In re-analyzing the CSE survey data in terms of this framework, 
Zucker defined a number of variables related to the characteristics 
of the district, the characteristics of the evaluation unit, the external 
role of the unit, and the internal role of the unit. In support of her 
propositions, she found, for example, that fewer than half of the 
units had internal roles in the selection of curricula or program 
materials, allocation of funds, facilities planning, collective bar- 
gaining, or teacher performance review. 
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Contribution and Importance 

There are manv in the evaluation communit> who may be rehic- 
tant to accept tlt^ole of evaluation units as mere conveyors of test 
data or evaluation data to outside agencies. However, studies such 
as those of David X1978), Alkin, Daillak, and White (1979), and 
Daillak (1981), corroborate Zucker's view that much evaluation 
activity is reg^ded as "paperwork" b> people withm the s>stem. 
Zucker\s analysis of the CSE survey data confirmed field work 
suggestive of this interpretation. 

The implications of these findings, however, are not totally neg- 
ative. They make clear the structural dilemma in which evaluation 
units are placed. It may be possible for units in some districts to 
differentiate their external and internal functions and intentionally 
separate these functions from one another. It may also be that as 
regulations and demands for compliance diminish in the coming 
years because of reconceptualization of federal and state involve- 
ment in local educational agencies, the evaluation and testing 
capacities developed within these units can be turned to produc- 
tive internal use. Or it may he that districts, once they have sat- 
isfied the signaling" function, can turn their attention to serving 
internal organizational needs. 
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Institutional Structure and 
Organizational Processes: The Role of 
Evaluation Units in Schools^ 



Historicall>, there has been a gradual but distinct trend toward 
, the vesting of more control in the federal and state government. 
Localities have not surrendered their autonom>, but have become 
increasingl> embedded in a multi-tier system, where state and fed- 
eral control and funding of local organizations has risen substan- 
tially in the past twenty years (see Zucker, 1980). While this trend 
^ is apparent in education, local school districts have been more suc- 
cessful than most public organizations in retaining their autonomy 
(Kirst, 1970; Meyer, 1979). , 

Schools are crucial to the pursuit of central societal goals, and 
therefore have be^n subjected to institutional definitions of what 
is proper educational procedure and practice. At the federal, state, 
and local level, school performance is monitored, evaluated, and 
subjected to Overall assessments. With the implementation of Title 
I of the Elementary and Secondary Education Act ^ 1965, local 
school districts were required to evaluate their performance. Fur- 
ther pressure for evaluation is rooted in the increasingly heavy 
dependence, especially in some states, on other ty pes of extra-local 
funding for school districts. These sourc^es of funding increase the 
legitimacy of external demands for assessment, local district needs 
for evaluation to improve program performance have largely been 
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eclipsed bv the need for accountafcijity- to funding sources. The 
current "raticJnal view** of schools held by these outside soiyces 
"ses a problem for schools because of the legitimate authorit> the 
^^deral and state evaluation requirements have over internal school 
practices. "H>per-rationalization** (Wise, 1977a and b) could be 
fgnored were it ndt for the concomitant attempt b> each separate 
funding bod> to enforce this view of educational process. — ^ 
Conformit> to institutional rules can be seen as the schools' 
raison^d etre. Man> of the processes in schools involve compliance 
to external definitions of what schools should do, and resporuJing 
to external mandates for evidence of such compliance. Especially 
as state and federal lunds have increasingly been allocated to local 
public schools, these external demands fo1v evidence have become 
more well defined and comple>^. Largely! in' response to these 
deijiands, local public school districts have established evaluation 
units. Such units are not required and are generally not funded 
directly b> federal or state soilrces but the requirements for eval- 
uation data (coupled with grant and special program evaluation 
requirements and h^fcCfrep^Ktihg requirements in some states) 
make it conveniem for school distfricts^to establish an evaluation 
unit. Evaluation units generally serve io centralize and simplify 
data collection and dissemination. It is not surprising that the 
niimber of evaluation units in school districts has iiterally mush- 
roomed since 1965. About a third of all s^ool districts with over 
5,000 students have evaluation units. Fully 85 percent of the eval- 
uation units have be^en established since 1965 and over half since 
1970 (Lyon, 1978, Lyon, Doscher* McGranahan, & Williams, 
1978). V . ' * 

This chapter develops a theory which emphasizes the signal 
nature of evaluation units. Depending on the^egree to which a 
.school district exists in an institutionalized environment, the- 
env ironment serves as a s6urce of suppojt {including funds) for it. 
As a result, Jthe envirgnment (state and federal government) exer- 
cises legitimate control and surveillance over the district's activi- 
ties and performance (for a general discussion, see PfefFer & 
Salancik, 1978). From this 'perspective, the primary function of an 
ev aluation unit is* to produce signals firom the school district to this 
external environment. Hence, most of the information produced 
by the evaluation unit is in response to accoufitability require- 
ments generated at the federal and state levels. Little of the infor- 
mation is used, within the organization (school district), though it 
may be disseminated within it (see David, 1978, for further sup* 
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port of this argument). Further, the evaluation unit generall> has 
a limited internal function (e.g., does i)ot affect district policies or 
other units), with little connection between evaluation unit staff 
and instructional staff. In other words, the evaluation unit serves 
to legitimated what the school district is doing b> reporting it (gen- 
erally in fav orable terms) to external sources of funding and control 
to which* the district is legitimately accountable. 

A general theoretical explication of the sources and conse- 
quences of institutional organizational environments is developed 
in the n^xt section. Then the concepts are applied more directly 
to evaluation units in school districts. Some central hypotheses are 
tested using data drawn from CSE s national survey of evaluation 
units (Lyon, Doscher, McGranahan, & Williams, 1978), combined 
with measures of federal and state funding and of control in.school 
district organizations. 

SOURCES AND CONSEQUENCES 
.OF FORMAL STRUCTURE 

In schools, as in other large-scale organizations, formal organi- 
zational structures arise, often not clearly or closely related to the 
actilal activities (i£jj«r organization (cf. Gouldner, 1954). This 
formal structure ij» fundamentally a blueprint for organizational 
activity. The organization chart, with its listing, of major officers, 
departments, and programs, details the organization s formally 
defined^tructure. These elements are linked by rationally defined 
coi[inections l)etween activities, impersonally embodied in the 
organization's explicit goals. 

Much of modern organization theory has been concerned with 
exploring the sources of formal structure in organizations (Scott, 
1975). Research thus far has investigated formal structure which 
arises primarily from problems of^coordination inherent in the core 
technology (e.g.. Woodward, 1965, Mqh^971, Comstock & Scott, 
1977, Billings, Klimoski, & Breaugh, 1977). In contrast, the 
emerging theory .of institutionalization (Zucker, 1977, Meyer & 
Rowan, 1977) has stressed the role of social definition, of "myth," 
in determining, organizational structure. Independent of the core 
technology, or "lopsely coupled" to it (see Weick, 1976, Meyer 
& Rowan, 1^77), formal structure created by social definition 
ser\e^ to legitimate the organization. The organization incorpo- 
rates elements of structure, such as evaluation units or affirmative 
action officers, which reaffirm organizational conformity to the 
externally imposed definition of what is legitimate. 
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This body of work on the sources of structure, whether task 
oriented or institutional, has treated the origin of formal structure 
ds largely outside of the organization's direct control: the structure 
is determined either b> the technolog> or b> the institutionalized 
environment. It Is import^mt/to note that wider societal definitions 
are involved in either case. The application of appropriate tech- 
nology is defined in terms of "state-of-the-art/' and organiza- 
tions are expected to adopt innovations defined as advances in 
technological development. Institutionalized procedures, certified 
professionals, and programs of action involve actual organizational 
activity, but at the same time incorporate societal defioitions of 
proper practices even when conformity conflicts with efficiency 
^criteria (Me>er & Rowan, 1977). Regardless, then, of the particular 
advantage or disadvantage for the task performance itself, wide- 
spread definition of a technology, procedure, or division of depart- 
ments ds rational,^ii»d legitimate leads to organizational adoption 
For example, earl> adopters of civil service procedures exhibit 
characteristics which indicate their need for more formal personnel 
procedures, while cities adopting these procedures later in the 
process do not, but rather adopt them simply on the basis of their 
widespread legitimacy (Tolbert & Zucker, 1980). 

But the organization itself should be recognized as an important 
di^terminant of its own location in the wider environment. Orga- 
nizations, whether technological- or institutional, are not simply 
passive captives of their environments. Their role in regulating 
environmental effects b> developing boundary maintenance func- 
tions, domain definitions, and other mechanisms of control has 
long been recognized in organization theory' (see Thompson, 1967). 
Most fundamentally, however, organizations seek to construct 
* their own environments, not sij^ply manage preexisting environ- 
mental constraints (e.g., Pfeffer's study of organizational merger, 
1972). For example, organizations may define their appropriate 
institutional location (Dowling & PfefFer, 1975). Training institutes 
wish to define themselves as educational institutions, thereby- 
gaining access to .societal resources such as tax write-offs and G. I. 
Bill funds, although their functions, in fact, parallel personnel 
agencies more closely. 

The foregoing argument can be summarized as follows: 

Projwsition I. Organi/^itions^seek to define their own location*in the 
wider insUtiitional/technologieal environment. 
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. Organizations define their location along a continuum, raaiging 
from U) narrow 1> construing their activities as technical and refusing 
societal resources (e.g., training stipends for workers) which would 
broaden the societal relevance of their goals, or (2) broadly con- 
staling their activities as fundamentally societal and' accepting 
societal resources which reinforce that view. 

This statement differs from earlier work on organizations in a 
number of respects. Probablv the most important, from an oper- 
ational point of view, is that the degree of dependence on societal 
resources -is a crucial step in defining the organization as one 
serving societal interests, and thus existing in an institutionalized 
environment. Hence, public organizations are, a priori, operating 
in uistitutionalized environments, though the extent' to which 
obligations of the organization are nonnatively defined may var>'. 
Organizations which are not public, but which deliver services 
s^en as related to the public good, ma> also operate in institution- 
<Uized env«fcf)iiments. Increasingl>, all organizations, including 
profit-making firms (e.g., LcK-kheed and Chrysler), are being iden- 
tified as central to the common good, and hence as deserving of 
societal support.-^ 

A coroHarv of central importance, since organiziitions are bound 
by histor>' and convention, is: 

C(yrollanj Once an organiz<itional t>pe is defined as societal or tech- 
nicaK other new organizations will have to demonstrate that they 
are not of that type before redefinition can succeed. 

Social definitions of appropriate procedures and practices have, 
over time, the force of facts. For example; educational organiza- 
tions are firmlv. einl)edded in the institutional environment;' it is 
unlikely that an educational organiziition coukl successfully rede- 
fine its environment a.s technological, and so escape accreditation, 
certification, and other institutional requirements (Meyer & Rowim, 
1978). 

Consequences of Institutionally Derived Structure 

Little work has focused on the consequences of formal orgaqi- 
/iitional structure, whether derived from technological or institu- 
tional environments. (ieneralLy, structure deriv(?d from needs of 




2 1 airt grateful to Marshall Meyer for suft^;estin« this interpretation 
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core technolog) is presumed to have consequences ldrgel> internal 
to the organization, altered task organization, altered efficiency 
(Leifer & Huber, 197T). But structure derived from institutional 
requirements is thought to he de-coupled from technical activities 
and to affect, primarilv, survival rates of organizations (Me>er & 
Rowan, 1977). 

Generallv, wMe organizations in the more ^'rational ' approach 
to structure are sSen as interpenetrated b> the environment, most 
of these inroads are^ither cxjntrolled or buffered to prevent them 
from having an> direct influence on internal organizational func- 
tioning or structure (Thonripson, 1967). T>picall>, organizations are 
seen as autonomous, in most senses not directl> accountable to 
interests located outside the organization. While private and public 
organizations are continuous!) interpenetrated b> customers (e.g., 
purchasers of cars) or clients (e.g., students of schools), it has been . 
noted generally that these groups do not exert much control over 
the respective organizations (see Hasenfeld, 1972, on people-pro- 
cessing organizations in general). While in principle such groups 
exert control because the> must select the product or serv ice, they 
frequently operate within limbed choice ranges (one must have a 
car and, according to the amount which can be spent, may have 
a choice between tw o or three xnajor competitors) or are a **captive 
audience" (requirements for school attendance coupled with local 
school concepts ensure, at best, a restricted range of choice). 

External Control . ' 

The ke> to understanding the effects of the environment cr* 
internal organizational structure appears to be the locus, exte(^"^-^ 
and legitimacy of external control over internal organ iziitional pro- 
cess. In normal market environments, organiziitions are viewed as 
legitimately autonomous in pursuit of their own goals as defined 
b> them. In striking contrast to this, organizations in institutional 
environments are seen as properly accountable to societal inter- 
ests, and therefore as legitimately controlled b> societal agents. To 
put it more directly, external administrative and legislative control 
is thought to be desirable in order to ensure that such organizations 
are, indeed, serving the sociefal interests they were created to 
serve or perpetuate*' * 

Such external control necessitates the development of rules and 
pro<;edures, and also rather formal rules for evaluation. As Kaufman ^ 
(I960), has pointed out, control without direct supervision (which 
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has been termed "long range contror*) necessitates development 
\)f elaborate monitoring 'and evaluation sv stems and other control 
techniques, such as professionalization, to maximize "self-regula- 
tion." It should be noted that all public organizations can legitN 
matel> be held acct)untable, since the> are ostensibl> organizecl 
for the public good. It is not the case, however, that all are actually 
so controlled. For example, when s^precy is essential (CIA, FBI) 
such control is relinquished. Further, some control is thought to 
be legitimate in private industry to the extent that the common 
good is potentiallv involved (e.frv^ railroads, air transportation, 
"vital resource" industries suchLiis coal and steel). • 
A second proposition can now4)e stated: 

Proposition 2 The greater the extent to which an organi^tion exists 
in an institutional environment, the greater the degree of legitimate 
control and demands for accountability coming from the relevant 
environment. 

The environmental control and demands for accountability are 
problematic to the extent that direct surveillance of organizational 
performance cannot be performed. As long as control is local, 
direct evidence can be obtained. But when funding and control 
come from extra-local sources, as in the case of school districts, 
accountabilitv becomes more problematic. Effectively, the orga- 
rn/atiou is recjuired to become "self-evaluating" (\Vildavsk> , 1972), 
and to transmit the information obtained to the external sources 
of funding and control. A third proposition, then, is: 

Proposition 3. If legitimate control and demands for accountability 
are extra-local, then orgiiniziitions will be required to perform eval- 
uation (or, minimally, monitoring) of their Qwn activities. ^. 

At the same time, however, the support and funding the orga- 
nization requires may be contingent on the results of the evalua-. 
tion. One major consequence of this is the creation of internal 
orgaruAitional uncertainties and inconsistencies. Loose coupling of 
administrative structure from the rest of the organization (and 
activities from each* other) may, instead of being dysfunctional, be 
the most rational strategy for maximizing production of positive 



^O'Shea discusses differences l)etween evaluation and monitoring elsewhere 
in this volume 
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evaluation results. Similarl), subunits responsible for responding 
to external demands for accountabilit> will be loosel> coupled to 
subunits performing other activities. This point will be developed 
more fully below. 

Formal Structure and Signaling 

Much as individuals use signals in negotiating the labor market 
isee Spence, 1974), organizations use signals to provide informa- 
tion to their environments. Organizations develop tasks, programs 
and goals, and even subunits to signal their intentions to the 
environment (on the latter point, see M. Me>er, 1980). Depending 
on the control which the environment has over the organization, 
the organization will proliferate formal structure which produces 
signals demonstrating accountability to the relevant environment. 

In brief, thu.s: 

Proposition 4 To the extent that the environment has legitimate 
control over an organization, the organization will produce structure 
which serves a signaling function. 

Even organiz<itions which are largely autonomovis from' the 
institutional environment ma>, under some conditions, be suffi- 
ciently constrained b> it to produce structure which serves a sig- 
naling function. A ca.se in point is the widespread existence of anti- 
trust departments in corporations. However, organizations more 
deeplv embedded in the institutional environment (Zucker, 1977) 
have elaborated man/ aspects of formal structure-j^hich produce 
signals demonstrating accountability — evaluation units in schools, 
designed to transmit infonnation concerning the (juality and degree 
of task focus, and "management information .systems" (MIS units) 
in local employment and training programs, designed to transmit 
evidence of internal competence in proces.sing clients. — 

Formal structure which serves signaling functions is, in many 
respects, ecjuivalent to boundary personnel. It represents the 
organiAitioii to the wider environment, signals its adequate per- 
^ formance as a means of generating additional resources, but has 
* little internal re.spon.sibility (see Zucker, 1979)."* Formal .structure 



••Its responsibilitv can be defined as "eiivimnmeiital iiian.i^emeiit," with few 
internal responsibilities 
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which serves largel> boundary lunctions appears loosel) coupled 
to the rest of the organizational structure precisely because it is 
not directl> responsible for, or not focused on, internal task per- 
formance. Hence: 

Proposition 5. To the extent that components of formal structure are 
designed to send signals to the wider institutional environment, 
they are loose!) coupled to internal organizational performance. 

Institutional Structure and Evaluation Criteria: 

What, then, of the content of the formal structure .and the 
activity it generates? First, the content of formal structure created 
ds a response to non-institutional, technological environments is 
primaril) internal and task-focused, while that created as a response 
to institutional environments is primarily externally focused. Hence, 

Proposition 6 Formal structure generated in response to techno- 
logical contingencies is concerned with internal organizational func- 
tioning, while formal structure generated in response to institutional 
contingencies is focused on external representation of organizational 
fiinctioning. 

Second, the actual activities generated by the formal structure 
will be different. Organizations operating in institutional environ* 
ments, as schools do, define their scope of activity to require being 
It^ss and less certciin about more and more (emphasizing ambiguity 
and uncertainty), while organizations not so externally accountable 
define their scope of activity to require being more and more cer- 
tain about less and less (emphasizing control and certainty). Schools 
(and similar organizations) need to emphasize' the uncertainty/ 
ambiguity because they are held externally accountable to different 
(and changing) standards of what is institutionally proper. Business 
organizations are not held accountable externally to the same 
degree and the external standards which dt) apply are more uni- 
form (e.g., arfti-trust regulations). 

Further, the greater the interpenetration of the local organizii- 
tions by institutional demands and resources, the more ambiguous ** 
and uncertain activities are presented; 

Pro;>(lvi/ion 7 The greater the extent to which an organization exists 
m an institutional environment, and the external evaluation there- 
fore becomes more critical, the more the organization stresses the 
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dmbiguit\ of evaluation cntena in order to make external evaluation 
more problematic. 

An important consequence of ambiguous criteria is that the orga- 
nrzation becomes less able to evaluate its own performance. For 
example, before schools were heavil> dependent on external (extra- 
local) support, validation, and resources, the> presented them- 
selves as producing ratlter specific training, evaluated on grounds 
of efRcienc>, ^schools moved into a more institutionalized envi- 
ronment, the> defined their\tasks as increa3ingl> ambiguous and 
uncertain, best evaluated on gfqunds of certification (see Me>er 
& Rowan, 1975;* compare Callahan, 1962 to Tyack, 1974). 

Organizations in institutional environments use the evaluation 
of task performance as a signal, alterable and ambiguous, to indi- 
cate compliance to institutional rules, not to guide adjustments in 
task orj^anization. Organizations in technological environments use 
evaluation of task performance as an index of how well the> are 
performing, as clear, unalterable evidence which can be used to 
guide adjustments in task organization.^ Characteristics of the task 
itself determine the extent to which it can be redefined as an index 
or a signal. Teaching, for example, is more ambiguous than auto- 
mobile assembl> (for a general discussion of activ e and inert tasks, 
see Dornbusch & Scott, 1975). Basicall>, it is asserted that orga- 
nizations in institutional environments will attempt to define their 
task performance in a different wa> than organizations not held 
accountable to societal interests. Fundamentally: 

(1) Organizations will act to ^produce and/or apply largely 
ambiguous criteria for task performance when societal sup- 
^ port and funding depend on evaluation: 

(i) They .will define the environment as more Variable, 
increasing; tHe soope of activities and making more aspects 
of the environment appear relevant to their tasks (e.g., 
range of pupil characteristics); 



*Spence (1974j «Lssumes that education ib a signal, that ib, an alterable observ- 
able characteribtic. while race is an index, that is, an unalterable observable char- 
acteristic. However, stKiulogists would argue that education will in most instances 
serve as an index it hough affected b> individual choice j the employer in fact 
assumes it is fixed at the current Jevel), while race is not uniformly an index, as 
the phenomenon of "passing" clearly illustrates. However, the willingness to 
identify and rely on signals or indices is of interest here. 
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(b) They will define tasks as more ambiguous and diflFicult 
(e.g., how can "good" teaching be defined, let alone reg- 
ulated?); and 

(c) The) will use evaluation information primarily as signals 
to the external environment. 

(2) Organizations act to reduce, and/or render inapplicable, 
institutional rules when evaluation control along with 
resources are centered in the local organizatiohal units (or 
when resources are controlled b> a large and difiuse "publiq"). 
. (a) They will define the relevant environment to make it 
seem more constant by reducing the scope of activities 
and range of relevance (e.g., only serve one small seg- 
ment of the market); 
(b) They will define tasks as clear, limiting the scope of the 
task (e.g., not to ^produce a "good" product, but one 
which meets certain pre-set specifications); and 
(cj The> will use evaluation information primarily internally 
(it may, in fact, be secret or privileged), as indices of 
task performance. Finally, the status of the organiza- 
tional members of units responsible for evaluation will 
var>', depending on the degree to which evaluation 
serves a signaling function. Thus: 

Proposition S The greater the signaling function of evaluation, and 
hence, the more it is viewed as ambiguous, the lower the internal 
status of the members or units responsible for evaluation. 

When evaluation simpl> serves to legitimate an organization, 
and is defined as largel> ambiguous and of little value to the orga- 
niz<itional task performance, then the ijnembers or units which pro- 
duce it*will be given minimal resources and support. 

In the next section, the relevance of this general model for eval- 
uation units in school districts is established. 

THE ROLE OF EVALUATION UNITS 
IN SCHOOL DISTRICTS 

It is clear that educational organizations do not operate in the 
rational technolog) -Oriented fashion that most organizational 
theorists have assumed (see especially Meyer, ,1977, Meyer & 
Rowan, 1977, 1978). Separate organi:^tional subcomponents and 
levels of authority do not articulate well with one another but, 
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instead, operate relatively autonomously with little control exer- 
cised (Weick, 1976 and Cohen, March, & Olsen, 1972, provide 
goo^ discussions of these issu'es). The lack of control over central 
task processes has been labeled loose coupling. The primary reason 
for the presence of loose coupling in educational institutions appears 
to be that the> function in env ironments whicK demand conformity 
to institutional rules rather than to technological perfection or 
mnovation, with some subunits largely engaged in producing sig- 
nals which indicate to the wider environment that the school 
is accountable to societal interests. As Meyer (1977, pp. 4-5) 
puts it, 

. . . the technical organization faces in toward its technical core and 
turns Its back on the environment, the institutional organization 
turns its back on its technical core in order to concentrate on its 
institutional enviroilment . . . [the] crucial thing a school needs to 
do to survive is to conform to institutional rules — including com- 
munity understandings — defining teacher categories and creden- 
tials, pupil selection and definition, proper topics of instruction, and 
' appropriate facilities. 

District-wide evaluation units have appeared, largely as a 
response to increased funding and control at the state and federal 
level, to handle reporting, monitoring, and evaluation require- 
ments. On a ncitionwide basis, 85 percent of all evaluation units 
appeared following the implementation of federal evaluation 
^requirements accompanying Title I in 1965 (David, 1978). In the 
CSE survey, only 3 out of 259 districts (1.2%) reported that they 
have^no state or federally funded programs. State or federally 
funded programs are evaluated by the evaluation unjt in over 80 
percent of the districts, while in less thanJ4 percent is this task 
delegated to other district units or personnel (and in le^s thim 5% 
delegated to outside evaluators solely). 

The direct relationship between the resources coming into the 
district and control is^ indicated in the following quote from a 
director of a Title I program (David, 1978, p. 13): "This district 
will accept all the strings that go with the Federal money." There , 
is little doubt that the primary function the evaluations serve is to 
meet state and federal reporting requirements. As one evaluator 
explicitly states, evaluation is generally associated with accounta- 
bility, not with information useful in assessing strengths and weak- 
nesses of programs (David, 1978, p, 39): "I don t know whether 
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the test scores are useful as a basis for making changes in the pro- 
gram because I don't deal with the content of the program." 

Before beginning a more extensive anal>sis of the data on eval- 
uation units, collected in the CSE surve>, the central h>T)otheses, 
and their relationship to the propositions above, need to be stated 
explicitly. First, as the percent of societal resources increases, the 
amount of regulation and control increases (Proposition 2). Second, 
as these resources are increasingl> extra-local, the organization will 
proliferate formal structure (units, positions) responsible for self- 
evaluation (Proposition 3). Third, as the degree of institutionaliza- 
tion ^dextra^lpcal control increa^^>, the pejc^nt of ^plf-evajyafipn 
information used extemall> also increases (the signal nature of 
evaluation, from Propositions 4 and 6). Fourth, as the information 
IS iricreasingl> used extemallv, the unit/personnel responsible for 
evaluation increasingl> have minimal influence over internal orga- 
niAitional policies, and have little connection to actual organiza- 
tional tasks (loose coupling of unit/personnel from P^roposition 5). 
Fifth, as the degree of extra-local control increases, the ambiguity 
of the information is stressed and a narrow range of evaluation 
takes place (Proposition 7). Finall>, as the ambiguity and narrow- 
ness increase and as the information use is more external, the 
status of the unit or personnel responsible for evaluation decreases 
(Proposition 8). o 

, THE TEST 

Since the analysis is preliminary at this point, full discussion of 
the results awaits additional work with the full data set. For now, 
a brief discussion of the CSE sample, the variables created for this 
analysis, and a very preliminary presentation of regression results 
serve to provide the framework for further study. 

The Sample 

There iu-e two basic survey samples, as described earlier in the 
monograph s Introduction. First thpre was a postcard survey con- 
cerning presence or absence of evaluation units in a nationwide 
sample of school districts. Second, a detailed survey instrument 
was sent to the heads of evaluation units in all districts having 
10,000 pupils, and to a 50 percent sample of districts having 5,000 
to 9,999 students. While the 263 respondent districts varied widely 
on man> of the characteristics measured by the survey instrument, 
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there is little information on the characteristics of those districts 
which do not have evaluation units, Table 1 exhausts the infor- 
mation available. 

Variable Definition 

There are several sets of variables crucial to this analysis. First, 
organizational characteristics are used largaly as given in the orig- 
inal data; (1) The staflF and budget of the evaluation unit (EU Staff, 
EU Budget), and (2) the Budget and assessed value of the entire 
district (Dist. Budget, Dist. Value). Second, the extent of local/ 
external resources is defined in terms of the percent white (highly 
negatively correlated, in turn, with fi-ee lunch provision and low 
student performance, both also good predictors of the inflow of < 
federal and state funds). Third, internal role of the evaluation unit 
is assessed by a simple summing of the number of school policies 
the evaluation unit is said to effect (Table 2 presents these mar- 
ginals). Fourth, tight coupling is defined in terms of the district 
having some general guideline affecting instructional activities. 
Fifth, the range of activities the evaluation unit engages in is 
defined as a simple sum (EU Act Range). s/f 

The largest class of v^iables created are those which measure 
internal/external emphasis in terms of time the evaluation unit 
devotes to each and in terms of the use of evaluation information. 
Since construction of these measures is complex, fuller discussion 
is deferred until the analysis is more complete (some of the mea- 
sures will be reconstructed to better reflect the underlying vari- 
ance). Basicall>, each measure is constructed so as to either weight 
the relative use (Test Use. Ext is a ratio of external to internal use 
of both criterion- and norm-referenced tests) or sum across a 
number of categories of internal/external (e.g.. External Use). 

Resuks 

Evidence for the role of external funding/control in the creation 
of evaluation units can be found in the distribution of evaluation 
units across states (Lyon, 1978). While in most states roughly a 
squarter to a third of the scho(51 districts 'have evaluation units, in 
states which have requirements for local evaluation and/or exten- 
sive state funding of schools, the percent having evaluation units 
is significantly higher (see Table 1). While the size of the district 
is important, with larger districts more likely to have both exten- 
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' sive external Rinding (especially in metropolitan districts) and eval- 
uation units, the variance across states is^t least as striking. While 
most states have no small (5,000 to 9,999 students) districts which H 
have evaluation units, states with extensive funding or evaluation 
requirements typically have evaluation units in a quarter to a third 
of their smalf districts. 



Table 1: Extent of State Regulation and Funding of Local 
Public Schools and Presence of Evaluation Units (N = 1321)* 



Regulation/Funding 


# Districts 
with EU 


Percent 


n" ' 


High State 


226 


45.0 


502 


Involvement 








Moderate State 


136 


25.7 


529 


Involvement 






290' 


Lx>w State 


47 


16.2 


Involvement 









♦ t'niverse of 750 districts with enrollments of 10.000 or more, 50% sample of districts 
with 5,000 to 9,999 students (573) Response rate of 100% for larger districts (n = 750), 81% 
for smaller districts (n=464) Table total n of 1321 reflects telephone follow up results 

Therefore, while all schools exist in an institutional environment 
iMeyer & Rowan, 1975), the degree of institutionalization does 
vary as a function of the differences in the amount of federal funds 
and regulations (depending on size, location, and minority/poor 
enrollment) and as a function of state differences in funding and 
control over local public schools. As Propositions 4 and 6 above 
state, under these conditions, as institutionalization increases eval- 
uation units will be more likely to be established, sfnce they con- 
stitute archetypical examples oC^rmal structure^ which signals 
compliance to external demands fo?s;jiccountability. 

To the extent that evaluation units serve as signals of compliance 
to external' directives and interests, their role in internal func- 
tionnig of the organization is predicted to be minimaKProp'^sition 
5). They are loosely coupled to the rest of the organization, but 
not by accident (Weick, 1976), they are established to serve as a 
boundary unit of the organization, to negotiate a complex institu- 
tional environment, not ao guid? the internal task performance, 
There is, of course, a complex relationship with the extent of fed- 
eral/state interpenetration of the local school, but as Table 2 dem- 
onstrates, it is fiiir to characterize tnost evaluation units as having 
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minimal internal roles. Onl> in the area of in-service training do 
over half of the ev aluation units neport having an internal organi- 
Ziitional role. 



. Table 2: Internal Role of EvaluatiofrtJnits (N=l227) 



Activity 


• Percent 


N 


Staff In-service Training 


66.5 


151 


Selection of Curriculum/ 


42.3 


96 


Program Materials 






allocation of Funds 


' 36.6 


83 


Facilities Planning 


33.9 


77 


Collective Bargaining 


14.5" 


33 


Teacher Performance 


9.7 


22 • 



Review 



There are frequentl> differences between the amount of effort 
an evaluation staff ma> devote tp generating information and the 
amount of and locus of iti use. For example, evaluation units 
devote more of their time to generating information intended for 
groups internal to the organization and less of their time to gen- 
erating 'information intended for external groups (federal or^tate 
agencies, parents). Howev-er, as predicted (see Proposition \6), 
actual use of the information generated is roughly equal externally 
and internally, depending on the particular measure. 

As external use of the evaluation information increases in impor- 
tance relative to internal use, the unit becomes increasingly loosely 
coupled to the rest of the organiziition, with organizational char> 
acteristics becoming less important in predicting bow effective the 
evaluation unit |)e;^eives itself to be in affecting use of its I^^ults. 
Following Proposition 7, tlie use pf evaluation research becomes 
more problematic, both externally and internally. Inconsistent, 
ambiguous relations between different measures of use becom*e 
app"nrent, and normal measures of performance of the evaluation' 
unit, such as the range of activities it performs, become decouplecU 
from the use of the information the evaluation unit generates. 

Table 3 presents the correlation matrix of all variables used, as 
well as means and standard deviations. The first main results are 
presented in Table 4. This tiihle is strongly supportive of the third 
hypothesis stated above. The higher the proportion of IcKal funds, 
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Table 3: Correlation Matrix 



Local/ External ^ 
Resources 
Inrole 

External t'se 
•I it^ternal.Use 

5 EUTime Ext 

6 EU Time. Int 

7 Test Use Ext 

8 EU Eval Fed 

9 feu Act Range 

10 " TiKht" Coijftin« 

1 1 Coasuit Budget 
12- EU Staff 

13 EU Budget 

14 Dist Budget 
15^ Dist Value 

Mean 
* . SD. ■ 
# 



z 


3 


4 


5 


6 


7 


8 


9 


10 


11 ' 


12 


13 


14 


15 


16 


007 - 


- 087 


079 


- 236 


266 


.030 


160 


-.087 


~ 025 


- 045 - 


-.373 


- 335 


-.268 


110 


.171 


X 


177 


162 


.163 


~ 202 


~ 038 


084 


.365 


.O53/^"^^^0l3V, 


.UOO 


.XJOO 




.063 


.153 


X 




346 


262 


- 292 


-.115 


- 110 


.179 


- 016 


.061 


uy / 


lift 




.086 


.102 


X 


X 


X 


,-.055 


068 


- 065 


^134 


.053 


124 


-.125 


.054 


.OOQs 


-.116 


. Vol 


.UOO 


\ 


X 


X 


X 


867 - 


Oil 


--^205 


. 136>^ 


.105 


- 032 


.085 


.085 


. 124 


— . 17o 




X 


X 


X 


X 


X 


.031 


.207 


-.177.^ 


- 106 


.056 - 


-.210 


- 154 


-.146 
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the less the time spent on evaluation for external sources and the 
less the external use of evaluation resultsrTime spent on internal 
evaluation and internal use is also negatively related, but not as 
strikingly. The evaluation unit staff is significantly smaller when 
local funds are a high proportion, though overall budget is not 
strongly related Revaluation unit or district}. Finally, as might be 
expected, district assessed value is positively related to extent of 
local support. j 



Table 4: Regression* of External and Internal Emphasis/Use of 
Evaluation Results and Organizarional Characteristics on the 
Extent of Local/External Resources (R = .21) 

Unstandardized Standardized 
Regression (Standard Regression 
Coefficients Errors) Coefficients F-ratios 



ExternalJJse 
IntemaTUse 


-1.17 


(3.63) 


-.04 


.10 


-.55 


(4.62) 


-.01 


.01 


•'EU Time. Ext 


-2.13 ' 


(1.46) 


-.46 


2.14 


EU Time: Int 


-1.61 


(1.42) 


-.37 


1.30 


EU Staff 


-.81 


(.55) 


-.41 


2.20 


EU Budget 


-.01 


(.01) 


-.06 


.09 


Dist. Budget 


-.01 


(.05) 


-.02 


.01 


Dist. Value 


.01 


(.01) 


.14 


1.58 



(Constant 1.39) 



Table 5 presents some strong results on the internal role of the- 
evaluation unit. As Table 2 abov e shows, mo it evaluation units play 
minimal roles in formulating internal schoou policy, even on issues 
directKy related to the evaluation function. What is striking in 
Table 5 is the strong relationship between a performance measure, 
EU Act Range, and internal role. Whilfe rmds strong, the pattern 
of external/internal ti|[ne and use is equally striking. Internal role 
is positively related to external use/time but negatively related to 
internal use/time, as the second hypothesis above predicted.. As 
might be expected, tight coupling in the district is directly related 
to an internal role for the evaluation unit. Finally, organizational 
characteristics are not strongly related to the presence of an internal 
role. 

Table 6 presents less consistent results. Rather than attempt an 
ad hoc interpretation at this point, the strongi^tionship between 
the amount of evaluation unit time spent for extehiaj purposes and 
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Table 5: Regression of External and Internal Empl^asis/Use of 
Evaluation Results and Organizational Characteristics on the 
External Use of Evaluation Results (R = .32) 



Unstandardized 
Regression 
CoefTiciertts 



^ Standardized 
(Standard Regression 
Errors) CoefTicients F-ratios 



External Use 
Internal Use 
EU Time: Ext 
EU Time. Int 
Test Use: Ext 
Local ^ 
EU Act Range 
"Tight" Couple 
EU St^fF 
EU Budget 
(Constant 1.82) 



14 


(.17) 


.08 . 


.57 


13 


(.23) 


-.06 


.31 


01 


(.07) 


.02 


.01 


05 


(.07) 


-.24 


.66 


09 


(.22) 


-.04 


.19 


13 


(.22) 


.06 . 


.34 


24 


(.08) 


.37 


9.48 . 


12 


(.17) 


.07 


.53 


01 


(.02) 




.33 


.00 


(.00) 


.06 


.08 



Table 6: Regression of External and Internal Emphasis/Use of 
Evaluation Results and Organizational Characteristics on the 
. , External Use of Evaluation Results (R #.26) 



Unstandardized 



Standardized 




Internal Use 
EU Time: Ext 
EU Time: Int 
Test .Use: Ext 
EU Eval Fed 
EU Act Range 
Local/Ext Res. 
"Tight" Couple 
Consult Budget 
EU Staff 
EU Budget 
(Constant 1.37) 



ession 


(Standard 


Regression, 




Icients 


Errors) 


Coefficients 


F-ratios 


39 


(.15) 


'.30 


7.10 


.05 


(.05) 


.3'6 - 


1.39/ 


.01 


(M 


.07 


.04 


.20 


(.14) 


-.15 - 


1.91 


18 


(.12) 


-.18 


2.29 


.01 


(.05) 


.02 ■ 


.02^ 


.00 - 


(.002 


.02- 


.03 


.08 


(.11) ' 


-.08 


.55 


.01 


(.01) 


.09 


.64 


.01' 


, (.02) 


-.10 


• .16 • 


.00 


(.00) 


.20 


-.86 



the external use of evaluation results provides some independent 
validation of the variables created. However, the strong negative 
relationship between the internal evaluation unit's evaluation of 
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fedcralvprograms (not outside consultants oj otfter school units) and 
external use is problematic. Further investigation of the external 
us^ measure seems appropriate, even though this table presents 
results which are largely supportive of the argument made here. 
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Notes on Grusky's ''Role Conflict 
and Ambiguity in Evaluation'' 



Introduction 

It has long beeu recognized that environmental factors affect 
organizational functioning, but within the past decade organiza- 
tional researchers have rapidl> increased their understanding of 
the elements and dynamics of this process. In his chapter, Oscar 
Gnisk> , professor of sociology at UCLA, notes that **theorists in 
the fields of contingency, resource dependence, ecology, political 
economy, and open systems . , . all emphasize in one form or 
another the impact of the environment on the focal organization." 
He cites HdWs Organizations. Structure, and Process (1977), Pfeffer 
and Salancik's The External Control of Organizations (1978), Aid* 
rich's Organizations and Environments (1979), Zald's Power in 
Organizations (1969), and Katz and Kahn's Social Psychology of 
Organizations (1966) as examples. Before presenting his re-analysis 
of the data collected in CSE's survey of school district evaluation 
units, Grusky discusses role conflict and role ambiguity as aspects 
of organizational functioning. 

Argument 

Gnisky svinterest is in both the school district as a parent orga- 
nization and in the evaluation unit as a sub-organization within the 
district. He is particularly concerned about the effects of theparent 
organization and of the sub-organization on individual role conflict 
and role ambiguity as perceived by the directors of evaluation 
units. 

He characterizes school district evaluation units as, typically, 
new within their parent organizations and possessing limited 
' resources. He describes the parent organizations' demands on 
these units for information as being somewhat beyond the units' 
capacity for adequate response. He notes^that the unit directors* 
roles can be characterized as boundary -spanning, that is, cutting 
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across internal lines of demarcdtion. Since research on "boundary' 
sp^ners" in other organizational settings provides reasons to expect 
that such individuals will experience both role conflict and role 
ambiguity, Xrusk> uses the CSE survey data to explore the effect 
of district and evaluation unit context on these two dimensions. 

Grusky defines "role conflict," following Rizzo, House, and 
Lirtzman (1970), as occurring when the occupant of a position 
encounters inconsistent demands and expectations from his or her 
environment. Grusky adapted six items from a Rizzo, House, and 
Lirtzman measure and had them included in the CSE survey, He 
found that on three items, more than^half of the evaluation unit 
directors selected the high role conflict responses agreeing that 
**I have received assignments without the manpower to conriplete 
them'', "I have to do things that should be done differently"; and 
. "I receive assignments without adequate resources to execute 
them." 

Grusky defines "role ambiguity," following Rizzo, House, and 
Lirtzman (1970), as occurring when the occupant of a position lacks 
appropriate role-related information as to duties, activity, and 
. allocation of time and goals. Most unit directors answered survey 

items related to this issue in ways to suggest that they felt little ^ 
role ambiguity. 

Grusky suggests that negative correlation between role conflict 
and ambiguity may be due to the amount of work done by the 
evaluation units: "The harder one must work . . . the less uncer- 
tain one is (or has time to be) about one's job." 

Grusky^s analysis of the survey data shows that there is consid- 
erable variation in expressed role, conflict and role ambiguity 
among evaluation directors. He explain.-; these variations in terms 
of three aspects of the district context and three aspects of the eval- 
uation unit's context. For the district, he looks at the variables of 
formahziition, size, and heterogeneity. For the evaluation units the 
variables are historv— the length of time the unit had existed, the 
' percent of district Kinds allocated to the unit, and the number of 
staff' in the unit. He finds that the three district variables— size, 
formaliziition, and heterogeneity— were significantly correlated 
with both ambiguity and conflict for individuals. For instance, the 
larger the district, the greater the individual's perception of role 
conflict. Grusky suggests that size may be related to organizational 
differentiation which in turn leads to lack of consensus and, there- 
ford, role conflict. For the unit variables he finds a slight negative 
correlation between history^ and ambiguity, that is, the longer the 
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unit had been in existence,, the less ambigujt> was experienced by 
the director. Gruslcv finds nonsignificant relationship between the 
unit variables and role conflict. - 

In his re-analvsis of the CSE data, Grusky is also interested in 
the extent to which a relationship existed between perceptions of 
role conflict and ambiguitv among unit directors and those who 
were reported as users of evaluation unit data. When the evalua- 
tion director reported both superintendents and principals as data 
users, he or she also reported experiencing little role conflict. This 
mav indicate ^hat these evaluation directors saw themselves as per- 
forming the jobs expected of them b> the clients most important 
to them. However, when the evaluation head reported a wide 
range of consistent users, he or she also reported some role 
ambiguit> The ambiguit> ma> .have resulted from the diversity of 
client^ served or from the skepticism of some of these cii^ts 
toward data from evaluation and testing. \ 

Contribution and Importance 

The position of director of an evaluation unit is not an eas> one.j 
Such individuals appear to experience high levels of role conflict. 
Grusky interprets the concomitant finding that the> also experi- 
ence low levels of role ambiguit> as meaning that the> have little 
power within the district. 

Grusky 's chapter points up important although hitherto largely 
neglected areas of inquiry for the researcher interested in evalu- 
ation utilization. The peripheral position of the evaluation unit 
within the parent organiz<ition, as well as the inconsistent demands 
and expectations placed on the directors of such units, suggest that 
evaluative activities, iio matter what their intrinsic merit or worth, 
mav often be ineffectual. If school districts, in fact, wi.sh to increase 
the use of testing and evaluation data internally, then the> must 
become aware of the intra-organizational dynamics contributing to 
role conflict and role ambiguit>. Although some factors, such as 
si/e and lictcrogeneitv of district, mav not be directly controllable 
b> district management, others, such as resources or clear speci- 
fication of evaluation unit clientele, may be. 
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Role Conflict and" Ambiguity Among 
School District Evaluation Unit Heads* 

Oscar Grusky 
University of California, Los Angeles 

This chapter seeks to explain variation in conflict and ambiguit> 
among the national sample of directors of school district research 
and evaluation units which formed the basis of CSE's study of these 
units (Lyon, Doscher, McGranah^n, & Williams, 1978). The 
chapter argues that variation in evaluation unit directors' role con- 
flict and ambiguity is a function of school district and evaluation 
unit characteristics since both sets of organizational features influ- 
ence the social context within which the director functions. 

School district research and evaluation init heads foce complex 
administrlttive problems. Their organizaticj^s are, typically, both 
new and small (Lyon, Doscher, McCranahXu. & Williams, 1978). 
The resources they have available, in part because of the unit's 
newness and size (which makes competition with other units dif- 
ficult), are scarce. At the same time, the demands placed upon 
them b> powerful persons and organizations in their school district 
and elsewhere are extensive and growing. Federal, state, county, 
and other units increasingly require information from school dis- 
tricts concerning the effectiveness of program functioning in spec- 
ified areas. 2 Moreov er, since the field of evaluation research itself 
is somewhat recent, unit directors-' background and training are 
frequently in other areas. This combination — inadequate resources, 
incr^'asing serv ice demand, and minimal job training— provides all 
the ingredients necessary for deep-seated role conflict and 
ambiguity. 

' I am grateful for the assistance of Catherine Lyon and for the support, finan- 
cial and otherwise, of the CSE staff, particularly Eva Baker and Ad nanne Bank 
This pro^ct \^as also partially supported by MMH (MN-14583). I am most 
appreciative of the research ,tssi^tance of Pamela Tolbert and the typing of Andrea 
\nM\one John Meyer and Mary Ann Millsap provided many helpful comments 
on the manuscript *> 

^For example, StufHeheam, et al (I97I) write. "As a response to outside pjj^s- 
sures, many school districts have iristalled or are now installing evaluation units" 
(p. 268, emphasis added). 
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Increasingly, organizational theorists are recognizing the signif- 
icance of environmental factors on organizations. Perrow (1979) has 
referred to this emphasis as a "new wave-gathering force." Theor- 
ists in the fields of contingency, resource dependence, ecology, 
political economy, and open systems (Hall, 1977; Pfeffer & Sal- 
ancik, 1978, Aldrich, 1979; Zald, 1969; Katz & Kahn, 1966) all 
emphasize in one form or another the impact of the environment 
on the focal organization. This emphasis is particularly important 
to the understanding of leader behavior in newly-created boundary- 
spanning organizations Mldrich & Herker, 1977) such as research 
and evaluation units. The research evidence suggests that boundary- 
spanners experience high levels of role conflict (Organ, 1971, 1976; 
Organ & Green, 1972; Adams, 1976; Miles, 1976). The evidence 
IS less conclusive regarding the relationship between boundary- 
spanning and role ambiguity. The mbst systematic study of ambi- 
guity is that of March and Olsen (1976). March and his colleagues 
not ortly view ambiguity as closely linked to the choice process but,N, 
in addition, assert that it is endemic to public and educational 
organizations (Cohen & March, 1974). In a recent review of role 
conflict research Whetten (1978) observed that ". . . what is sig- 
nificant about the literature on boundary spanning is the noticeable 
lack of interest in systematically exploring the sources of role con- 
flict." With the exception of the March and Olsen study, the same 
could be said about research on role ambiguity.^ 
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APPROACH AND HYPOTHESES 

Although role conflict and ambiguity are related (see Kahn et 
al., 1964), they are not identical.' Conflict arises from the quality 
and quantity qf demands placed on persons while ambiguity refers 
simply to perceived uncertainty. The relationship between these 
variables is largely unexplored. March and 01$en (1976) claim that 
"individuals find themselves in a more complex, less" stable, and 
less understood world than that described by standard theories ot 
organizational choifce; they are placed in a world over which they 
often.have only modest control" (p. 21). Ambiguity in educational 

1 Vlarcli and Olson ( 1976) refer to four types of ambiguity the ambiguity of 
attention ambiguity of understanding, ambiguity of history ,,and the ambiguity 
of organization The ambiguity measurf we used doesn t begin to do justice to 
the richness of this typology 
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organizations, especiall> among leaders in this t>pe of social s>stenn, * * 
is the name of the game. As for role conflict, the pioneer empirical 
stud> of the phenomenon b> Gross, Mason, and McEachem (1958) 
v^ds of school principals. Giv en the newness and instabilit> of 
school evaluation units both role conflict and ambiguit> shoulc| be 
extant in our sample. 

Contrar> to the standard portrait of schools as unsuccessful 
organizations, Me>er (1977) pictures them as highl> successful 
because the> have survived and even substantially expanded their 
resource base. The basis for their success is their conformity to 
society's institutionalized rules and the fact that they have become 
^'relatively decoupled from the technical work of instruction." 
Unlike business firms which carefully control their technical struc- 
tures, schools leave their instructional tasks relatively unevaluated 
and uncontrolled. Since evaluation units may be technically 
respoiisihle both for student testing and^ rev iewing instructional 
programs, we might expect that evaluation unit directors face far 
inure conflict and ambiguity than their counterparts in business 
firms. Their work has actual or potential relevance for the school's 
most fundamental tasks. 

Our approach emphasizes the preeminent effect of context or 
structure on the organizational subunit and thence on role conflict 
and ambiguity. Three school district variables influence the con- 
text within which evaluation units function, formalization, size, and 
heterogeneity. Each of the variables affects the extent of interest 
group pressures Itkely to be experienced by the unit director. 

"Formalization" refers to rules, its opposite is anomie. The more 
re{{ulated the district organization the greater the unit's adminis- 
trative control over uncertainty. The more formalized the orgatii-. 
,zation, the more protected the unit heafl feels (Gouldner, 1954, 
Miles & Perreault. 1976) and the less likely he or she experiences 
role conflict. Both district size and heterogeneity influence diver- 
^ sity of interest groups in the district. 

Kahn et al. (1964) note that persons in positions that link units 
are more likely to be subjected to conflicting requirements and 
pressures because they interact with persons who have competing 
goals and standards. However, formalization should relieve some 
of this conflict insofar aS rules closely specify task and goal respon- 
sibilities. A district's size and heterogeneity affect its political 
capabilities and its ability t() capture resourcejr from the society. 
While, on oneShand, large size and heterogeneity demand respect 
and hence enable districts to command greater amounts of 
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resources, on the other hand, the\ impl> more competing interest 
groups, the greener the diversity of interest groups in a school dis- 
trict, the more Tikel> there uill be conflict among them. H\po- 
theticallv, mterest group conflict m the school district should lead 
to role conflict and ambiguity among evaluation unit heads. 

The unit variables of concern are history and resource availa- 
bilitv. Pfeff^r and Salancik il978) suggest that survival is the ulti- 
mate test of organizational. effectiveness and history is inextricably 
hnked to that concept. In the present study, history was measured 
by length of time the unit has existed, resource availability was 
measured specifically by budget and number of staff personnel. 
History and unit resources are substantially determined by deci- 
sions external to the focal unit. Pfeffer also proposed that persons 
have less effect on organizations than does the institutional context 
because selection processes ensure homogeneity among leaders. 
Leaders are seen as having little discretion, anyway, since the 
major impact on outcomes stems from resource availability and, 
in school districts in particular, this is generally outside the unit 
head s control (Leiberson 6c O'Connor, 1972). Since a unit director's 
uncertainty ma-inly revolves around resources, we might lilive 
anticipated that unit variables would have a -greater impact on 
ambiguity than on conflict. 



Conflict 

"Roles" are generally defined as sets of expectations about 
l)ehavi()r associated with e^^rganiziitional positions. Role conflict 
takes place when the cx;cupant of a position encounters incon- 
sistent demands and expectations. Four types of role conflict have 
been klentified by Rizzo, House, and Lirtz^nan (1970): 

1 Conflict between the focal person's internal standards or values 
and the defined role behavior ... 2.^ Conflict between the time, 
resources, or capabilities of the" focal person and defined role 
behavior . ..3. Conflict between several roles for the same person 
which re(|Uire different incompatible behaviors ... 4. Conflicting 
expectations and organiz<itional demands in the form of incompat- 
ible policies, conflicting requests from others, and incompatible 
standards of evaluation. 

Rizzo, House, and Lirtznian developed the factorially identifiable^ 
and independent measures of role conflict and ambiguity that we 
adapted for use in the present project. Six items with the highest 
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factor loadings were selected from their larger set. The items, 
listed with percent agreement in our sample, were as follows (see 
the monograph Introduction for a description of the sample 
selection). 
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Items 


% Agree or 
Strongly Agree 


I r<*ceive ussignnients without the ni an power 
to complete them 


00 


I work under incompatible policies and 
" guidelmes. 


' 21 


I have to l)Uck a rule or policy in orclpU«**»w^^ 


"V 20 


carry out an assignment. ^•^^""""^ 




I receive* assignments without adequate 
res(nirc(»s and materials to execute them. 


^51 


\ have to do things that should he done 
differently. 


53 


I receive incompatible re(iuests from two or 
more people 


"27 







It should J)e noted that over 50 percent of the respondents in 
our studv selected the higli role conflict response in three of the 
six iteirp?r-Two of these tfiree were concerned with inadetjuate 
resources. 



Ambiguity 

"Role ambiguity " refers to the situation that takes place when 
the occuptmt of a position lacks tlnsappropriate role-related infor- 
mation. This occurs when the position fs not clearly defined or 
when access to needed infprmation is impeded (for example, 
because of the occupant s inexperience or because of the newness 
of file position in the organi/iition). SpeciQcally then, ambiguity 
refers to the degree of felt certainty regarding one's duties, 
authorit>, allocatiqn of time, and goals. To measure ambiguity the 
five items with the highest factor loadings were selected from the 
Ri/v.o.- House, and Lirt/man set. The items, listed with percent 
agreement in our sample, were as follows: 
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% Disagree or 


Items 


Strongly Disagree 


I feel certain about how much authority 1 


18 


have. 




I have dear, planned goals and objectives for 


12 


my job. 


30 


I know that I have divided my time properly. 


I know what my responsibilities are. 


10 


I know exactly what is expected of me 


20 



It IS apparent from the above that role ambiguity was less . 
common among director* than was conflict. A large proportion of 
the sample reported little ambiguity. Although we lacked com- 
parativefcata, these findings would seem to contradict March and 
Olsen's claim regarding the pervasiveness of ambiguity. Consistenl--^ 
with Ri/zi), House, and Lirtzman. we found a significant negative 
correlation between role conflict and ambiguity. (-r= - . 19, p<.001. 
See Table 1.) It mav be that conflict produces expectational clarity. 
That is, the harder one must work and the more corners that must 
be cut JJt) get the work done, the less uncertain one is (or has time 
to lx*7about one's job. 

The evaluation unit heads were, txpically. highly experienced 
and professionall> trained individuals. Sixt>-five percent he.ld the- 
doctorate, most usuall> in administration. elementar> or secondary 
education, statistics, or educational or ge^neral psychology. Almost ^ 
three out of ten had been school principaFs and over half had once . 
been elementary or second^iry school teachers. Very few (l4.4%) 
bad had an> formal course work in evaluation. No significant rela- 
tionship was found between taking such courses and role conflict 
or ambiguity. - 

One important ftinction of school district evaluation and research 
units is to provide information of value to school administrators. 
Most of these units report directly to the superintendent or through 
one interrnediarv. The job involves monitoring school programs 
inclirectlv and empha.sizes testing student achievement. 

School District Variables * 

Three variables were used, formaliziition, size, and heteroge* 
neitv. Hage and Aiken (1970) and Hall (1977) define formalization 
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as the rules and procedures organl^tions establish to handle con- 
tingencies. All unit heads were asked to report the extent to which 
there were written school board policies in six areas, student con- 
duct in classrooms, introduction oF instructional innovations, t>pe 
of curricular piaterial to be used, student conduct on, school 
grounds, instructional methods teachers use, and criteria used in 
evaluating student learnirfg. This was consistent with Pligh et al.'s 
il96S) definition of formalization as "the extent to which rules, 

' procedures, instructions^ and communications are written." A 
•^factor analvsis of the scale resulted in one factor (unnained) that 
e.xplained 39 percent of the' variance of the items. Average item- 
item correlation was .26. Cronbach's (1951) Alpha was* .68, indi- 
cating replicability and reliability.'' 

School districts, were classified by size into four groups, met- 
ropolitan districts (enrollment, 45,000 or more), large districts 
ienrollment, 25,000-44,999), medium districts (enrollment, 10,000- 
24,999), and small districts,(enrollment, 5,000-9,999). Existence 
of an evaluation unit was positively related to size. (Districts under 

, 5,000 students were excluded from the Study.) 

"Heterogeneity" referred mainly to the ethnic-racial student 
mix iri the district. The measure selected was percent of students 
eligibje for the nationwide free-lunch program. As Table 1 shovy,, 
this measure correlated significantly wifh percent White, percent 
Black, percent Hispanic,' and percent students scprii'ig in the 
bottom quartile. ♦ . , 

Xable 1. Correlations Among Indicators of District 
Heterogeneity 



Percent 
Black 


Percent 
Hispanic 


Bottom 
Quartile 
Students 


Percent 
Free Lunch 


Percent White - 77** 


-.47** 


-.6,3** 


-.76** 


Percent Black * 


-.12** 


.56** 


.69** 


Percent Hispanic ' 




.27** 


.2.5** 


Bottom Quartile 


< 




.66** 


Students 








* p ^ . 05 
**p < (X)]. 

Trsts of sigrtificante arc two-tailed 









upon request 
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Blau (19TT)v^efines heteroger\eitv as "the 'distribution of a pop- 
ulation among groups in terms or^ nominal parameter" (p. 9). He 
lists thirteen nominal parameters, sex, race, religion, ethnic afifil- 
lation, clan, occupation, place of work, place of residence, industry, 
marital status, political afRliation, national origin, and language. 
The greater the number of groups and the more evenly a popu- 
y lation is divided among them, the greater the,heterogen^t> . The 
free^lunch prograhi is based on willingness to participate. Using 
this measure as a heterogeneity index probably maximized the 
ethnic-racial mix as well as the mix on marital status, national 
origin, and language. On the other hand, it may well be associated 
with econumic homogeneity. Unfortunately, data were unavailable 
to ascertainthe association of the index with each of these variables. 



Evaluation Unit Variables 

V 

"History" referred to the length t»f time the unit was in exis- 
' tence. As anticipated, most wer^ new organizations. Ov^r one- 
third (35%) were five years old or less while 62 percent were ten 
years old or under. Only about one-seventh of the units (14%) had 
been in existence fifteen years or longer. 

Two indexes of resources were used. Monetary resources \vere 
measured by the unit's allocated percentage of the school district's 
yearly budget, hi general, the larger the nx\\i\ percent of the 
budget the greater the amount of slack resources (defined by Cyert 
& March, 1963, as the difference between existing resources and 
activated demands.). Personnel resources were determined by the 
number of full-time staff in the uniy In 23 units only part-time staff 
were employed, in 108 there was orily one full-time employee, and 
81 units ranged in size from two\o five full-time persons. The 
largest unit reported 90 staff members. It may be assumed that the 
larger the staff the greater the personnel resources and the mor((_^ 
slack. ' • 7 ^ , ^ 
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In Table 2 the means, standard deviations, N*s, and intercor- 
relations are presented for the principal variables used in the 
study. The findings show that the three district variables — size, 
formalization, and heterogeneity — were significantly related both 
to ambiguity and conflict, while the unit variables — history, budget, 
and staff — correlated with ambiguity but not conflict. 
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Table 2. Me^ns^ Standard Deviations, Ns, and Correlations of Dependent and Independent Variables 



Variable 


N 


Me|in ^ 


Standard -J. 
Deviation Ambiguity 


History 


EU Budget 


), 

EV Staff 


District Size 


District 
Formafization 


District 
Heterogeneity 


Conflict 


263' 


2.23 


69 ^ 


- 19*** 


- 004 ' 


- a3 


07^ • 


16^^* 


-.08* 


12** ' 


Ambiguity 


263 


2.96 


47 ^ 




-'07^ 


19*** 




03 


10** 


* 13** • 


History 

EU budget • 


249 
223 


68.33 . 
33.83 


10 36 ' 
42,68 






16** 


- 10** 
31*** 


16** 

' 007 


- 01 

- 04* 


- 08* 

06 


Staff 


262 


4 77 


9.94* 










44*** 


- 03 


25*** 


Distnct 


263 


251 


99 












.002 


.33*** 


Size 






















District • 


263 


1 95 


.42 • 














- 16** 


« Formalization 




















» • * 


District 


225 


24.74 


19 79 
















HeterogCDeity 






1 
















*p<.10 
**p < .05 
***p < .001 

Tests of significance are 2*tailed 
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For the evaluation unit variables, a slight negative correlation 
wa6 found between histor> and ambiguit>. The Fonger the unit had 
been in existence, the less ambiguit> was experienced by the 
director. A modest relationship was found between the two resource 
indexes and ambiguit>. Budgel and stafiF a\|ailabilit>, which not 
surprisingl> were positivel> correlated with one another, generate 
increased demands on the director. In March and Olsen s (1976) 
terms, slack provides solutions for problems and sufficient partic- 
ipants for each and ever> choice. The greater the amount of slack 
resources, the more problems for the director and hence the 
greater his/her uncertainty' as to how to resolve them. 

No significant relationships w^re found between th^ three unit 
variables! and role conflict. Two opposing hypotheses were j^os- 
sible. that new units would produce more role conflict than old 
onejr in that the former, being less institutiotialized, woultf be less 
able 'to reconcile incompatible demands and pressures, or, alter- 
nativelv^ that old units would experience'greater role conflict since 
the> had had more time to become knowTi, vvgulcl thereby gen- 
erate more demands from external units and> nence, experience 
greater pressures than new tinits. However, neither history nor 
slack engendered inconsistent demands and^ expectations for the 
director. Jt might have been anticipated that ^ack would increase 
the director s role -conflict since demands increase when more 
resources are available, that insofar as these demands outrun, 
resources, conflict rej^ults. This model suggests that the relation- 
ship between slaek and conflict is curvilinear. 

The bivariate relationships between school distdc^t characteris- 
tics and the dependent variables were, with one exception, 'statis- 
tically significant. District size was related to role conflict but not 
to ambiguity. Kahn et al. (1964) aho found a significant correlation* 
between size and role conflict. Size has been related to structural 
elaboration (Meyer, 1972) and to subgoal development (Dearborn 
& Simon, 1958), both of which are indexes of differentiation. Dif- 
ferentiation creates a lack of consensus which generates rgle con- 
flict for the administrator. 

Formalization was negatively related to role conflict which m<iy 
mean that rules act as intended in regulating expectations and 
enhancing consensus. However, formalization was positively related 
to ambiguity. That is, tTie more rules, the greater the directors 
uncertainty — obviously not the intended ^function of rules. It may 
be tljj^EUirge numbers of rules arrd policies are so cumbersome and 
complete that they induce uncertainty among heads of units. 
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Heterogencit) was the final context variable. Not surprisingly, 
It was signficantl> correlated with district size. This was reassuring 
since large metropolitan districts should be the most divers^ and^ 
small ones least diverse. Heterogeneity was positvvel> related to 
role conflict., This finding supports Thompson's (1960) theor> which 
asserted that heterogeneity of organization members generates 
role diversity which, m turn, causes organizational^nflict. Orga- 
nizations with heterogeneous populations develop numerous "latent 
roles ' which present complex management problems. The same 
finljing would be predicted b> political economy and resource 
dependence theory (Pfeffer & Salancik, 1978^ Zald, 1969): the 
greater the heterogeneity of the district, the, more diverse and 
extensive are competing groups. The more there are of such 
groups, all seeking to assert their interests, the more conflict 
experienced by the unit head. His or her task is complicated under . 
such circumstances as he or she. seeks to reconcile demands for 
information from teachers, community groups, parents, program 
^directors, the school board, principals, administrators and, in the 
' case of desegration, the courts. Heterogeneity was also positively 

correlated with ambiguity. One explanation is that heterogeneity 
lejids to increased needs for information input and for distribution 
of output^ The greater the number of sucJi demands the less certain 
the director is regarding duties, authority, time allocation, and 
objectives; hence, the greater the felt ambiguity.* 

The regression analyses were designed to tell us how much of 
the variance in conflict and ambiguity the complete set of inde- 
pendent variabltis explained. The regression equation used took 
the following basic form. Conflict = a + T^i (SchooH District Vari- 
ables) 4- ba (Unit Variabtes) + Ambiguity + Error. 

The independent variables were regressed in stepwise fashion, 
first on conflict and then on ambiguity, The district variables were 
entered first since presumably they were less controllable by the 
directors than were the unit variables. Tables 3 and 4 present the 
mam finflings.' The multiple R for the equations ranged from .18 
to .35 indicating that the independent variables accounted for only 
about 3 ^rcent ,to 12 percent of the variance in conflict and 
ambiguity. Obviously, this was not a great deal. The more con- 
servative adjusted measures, which consider the number of 
variables in the equation, reduced this amount further 

To suminari/e this section and the results thus far, we antici- 
pated that a selected set of school district variables and evaluation 
unit variables would explain role coTiflict and ambiguity among unit 

t \ 
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Table 3. Regression of School District and Evaluation Unit 
Variables on Role Ambiguity 





UnstandardiiSed 




Standardized 


iiiucpciiucii I 


n c b 9 1 uii 


ouinaaro 


ricgrcbbiuii 


Variable 

* 


Coefficient 


« Error 


Coefficient 


Heterogeneity 


:306 


.001 


.132 


Size 


-.233 


038 


-.051 


Formalization 


.158 


.078 


.144 


History 


-.308 


.003 


-.065 


EU Budget 


.155 


.0008 


.146 


EU Staff 


.454 


.003 


.105 


Conflict , 


-.128 


.048 


-.192 


(Constant) 


3.067 






Multiple R 


.350 






R Square 


.122 






Adjusted R Square 


■ • .088 




1 



Table 4. Regression of School District and Evaluation Unit 
Variables on Role Conflict 





Unstandardized 




Standardized 


Independent 


Regression 


Standard 


Regression 


Variable 


Coefficient 


Error 


Coefficient 


Heterogeneity 


.366 


, .002 


.105 


Size 


.138 


.057 


.200 


Formalization 


-.148 


.120 


-.090 


History 


.143 


.005 ' 


.020 


EU Budget 


.161 


.001 


.0001 


EU Staff 


.948 


• .005 


.014 


Ambiguity 


-.293 


.111 


-.194 


(Constant) 


2.888 






Multiple R 


.337 






R Square 


.113 






Adjusted R Square 


.078 







directors. In fact, we were able to explain only a small portion of 
the variance. We turned, next, to the problem of organizational 
influences on the use of evaluation information. The main question 
we sought to answer here was. Is there a relationship between the 
reported users of evaluation data and role conflict and ambiguity 
among unit heads? 
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Conflict, Ambiguity, and Use of Evaluation Data 

Lyon,* Doscher, McGranahan, and Williams (1978) provide 
extensive information from the CSE studv on what evaluation unit 
heads do. The> found that 95 percent of the unit directors ranked 
assessing student achievement as one of their most tim^-con- 
suming activities. Sevent>-four percent ranked it as the most time- 
consuming. Seventy -five percent of the directors claimed that 
testing was the major data collection method. Moreover, most unit 
heads reported that almost half their tim^, 46 percent, was>spent 
on earl> childhood and elementary education. The authors of the 
CSE report concluded that **. . . the survey and fieldwork confinn 
the continuing dominance of testing in all activities of evaluation 
offices ..." (1978, p. 100). 

David's (1978) intensive field stud> of school district use of Title I 
evaluations found that they . . do not primarily serve either as 
a means of judging the program or as a guide to program improve- 
ment" (p. v). This was so for three reasons. (1) Most programs were 
stable, onl) minor changes took place an^'way. (2) Typically, eval- 
uation results were received too late to be useful, and (3) Other 
factors, such as political demands, played a key role in program 
change. If these findings hold up when replicated then we must 
assume that these units mainly meet reporting requirements and 
do not play a significant pc^^rt in program change. This may be 
because there isn't a great deal of program change other than that 
which comes about as a result of extemall> -imposed legislation. 

Respondents in the CSE study were asked to identify the major 
users of their units' reports. It was found that the consistent users 
were program directors (62%), superintendents (60%), central 
off^ice staff (58%), and principals (52%). Only one-third reported 
teachers as consistent users. This was about the same percentage 
reporting federal and state agencies as users. It was evident that 
the units mainly serve the school administration. , 

As Table 5 demonstrates, role conflict was negatively corrected 
with service use by superintendents and/or principals, that is, 
those who did not report these parties as consistent users were 
most likel> to experience high conflict. This suggests that the 
closer the service ties between the evaluation unit head and the 
school superintendent and/or principals, the less conflict was 
experienced. In other words, the way unit breads reduced stress 
was b> accommodating to those who have administrative authority 
or influence. 
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Table 5. District Use of Evaluation Unit Data arid Role 
Conflict and Ambiguity 



Reported Consistent User Role Conflict Role Ambiguity 

Superintendent . -.20 (p = .02) H&lp = .02) 

Principals -.18 (p = .03) .12 (p=.10) 

Board members ^ J9 (p = .001) 

Parents or local citizen groups 

Teachers ^ -.21 (p = ;007) 

Central office staff -.15 (p=.04) 

Federal agencies ^ .10 (p = .003) 

State agencies 

Program Director ^ .13(p=.07) 

A different pattern was found for ambiguity. Unit heads with 
high ambiguity were more likel> to report superintendents/prin- 
cipals, program directors, Board members, and federal agencies as 
consistent users of their serv ices and less likel> to report teachers 
and central office personnel as users. It appears that the greater 
the range of perceived use of evaluation services, the more the felt 
ambiguity. Any type of administrativ e contact can generate uncer- 
tainty, but contact with those highly placed in the organiziition 
(such as superintendents, principals, board members) was associ- 
ated with high ambiguity while contact with lower level roles 
(teachers, central office staff) was associated with low ambiguity. 

Ambiguity occurs when shared role specifications are incom- 
plete—the officeholder is unsure of what is desired or how to 
behave. Unit directors felt most urtcertain when consistent users 
were principals, superintendents, board members, program direc- 
tor's, and federal agencies. Perhaps this was because these.offiicials 
rfot onl> have organizational clout, but also have little confidence 
in the test data the evaluation units produce. Their profound lack 
of confidence in'the units* major product was described and ana- 
lyzed by David (1978) who quoted several officials* telling criti- 
cisms of standardized test results;^ 

How can you evaluate when kids are starting at different places and 
developing at different rates? Means don*t mean anything. (Director) 



*0*Reili> amplifies these kinds of criticisms elsewhere in this monograph. 
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Individual diagnostic tools provide the basis for my judgment of 
program success, not the standardized tests. (Principal) 

If the standardized test scores are negative, it's okay because 
everyone buys the argument that they can be discredited. 

(Administrator) 

David (1978) also reported that teachers were critical of testing. 
However, unit heads* ambiguity was low when teachers were con- 
sistent users, probably because the> were less threatened by 
teacher criticisms. Being held in low regard by their superiors 
understandably carried more weight. 

CONCLUSIONS ^' ' 

Juvenal wrote in his Satires, "But who is to guard the guardians 
themselves?" while Plato, much less the realist, stated in the 
Republic, "What an absurd idea — a guardian to need a guardian.** 
Evaluation is a booming enterprise and evaljuation units in school 
districts are to be found in many districts of*substantial size. These 
units are conceived by some to be public guardians, data collec- 
tors, and assessors. 

This chapter argues that the social resource char'acteristics of the 
school district — that is, the external context within which evalu- 
ation units function, and the organization of the linit itself — are 
key sources of mformation about them and particularly about the 
amount of conflict and ambiguity confronted by the directors. Con- 
trary to expectations, school district and evaluation unit variables 
did not explain much of the variance in the directors' role conflict 
and ambiguity. However, our findings do suggest that evaluation 
unit heads fill a ver> difficult position in the school district and that 
a ke> source of their difficulties is that tfieir main output i§ not 
highly regarded b> their superiors^' StilP another problem sfems 
from the^fact that the> have limited contact with the programs they 
evaluate. These two problem's are related because if they had 
better contact with major school programs fhey would have access 
to information which could enhance their organizational position 
and power and improve the quality of their contribution. The 
directors* overall level of role* conflict was high while their level 
of role ambiguit) was low. Since power and ambiguity are highly 
correlated, this latter finding reflects their low power. It may be 
that the tasks of the directors are too well established and not 
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ambiguous enough. If evaluation units are to make a difference in 
school district innovation and functioning, they jrnusit involve therii- 
selves closely in classroom activities and related programs. Although 
this would produce greater uncertainty for the directors it could 
a^o I\elp make their -evaluation tasks considerably more meaningful. 
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Notes -611 O'Sheas "Evaluaftion Units 
Problems and Possibilities" 

• * ■ • 

Introduction * , ^ ^ 

Al] organisations reflect at some times on sifch questions as "Are 
we successftil in what we are doing?" "Are our techniques working 
properly?" "What is in need of change and improvement?" A 
variety of responses and methods have been developec^ in the pri- 
vate and public sectors to aj^swer these questions, rang;ing from 
leaders' guesses or hunches, through monitoring and control pro- 
cedures, through market research, and finally to elaborate man- 
agement information systems. 

In education, a recent response to questions of effectiveness has 
been the emergence of the field of educational evaluation. Over 
the years evaluation has developed from an offshoot of educational 
research,to an established field with its own unique problems and 
procedures. A*current manifestation of that growth, and ah iinpor- 
tant milestone in evaluation's history,* is ttie recently published 
Standards for Evaluations of Educational Programs, PrQjects, and 
Materials (Joint Committee on Standards for Educational Evalua- 
tion, 1981) which attempts to provide a common language and set 
of procedures for evaluation practitioners and theorists. 

But a fundamental problem plagues educational evaluation. 
^Evaluations use uncertain methodologies .to study technically 
uncertain'activities that take place in uncertainly understood con- 
texts. By this we mean that evaluations, wTiich are almost always 
^carried out in a field setting, are necessarily limit^ in the ^in'ds 
of researcli design that can be employed. Many, proT^ably most, 
programs and projects that are evaluated h^ve many factors oper- 
ating at the same time that can influence, usualJjM^ indetermin- 
able ways, the program or project's outcome. 

Given these serious constraints, what is the maximum contri- 
bution, in education, that evaluation 'can make towards answering 
the age-old questions about organizati^Jnal effectiveness and 
improvenient, with which we opened these conwnents? It is 
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important for the evaluation cx)mmunit> to address these questions 
so that work can be directed towards reducing the uncertainties 
and channelling energy into more achiev^able directions. 

/^rguni^nt 

In his chapter, O Shea explores the built-in limits of evaluation,' 
as it is currently practiced, ^ an aid to educational decision 
making! While O^Reilly Examined decision makers and their con- 
texts, O'Shea looks at the inherent assumptions underlying edu- 
cational evaluation itself and points out that such evaluation is now 
frequently confined to use as a monitoring device rather than as 
an analytical tool, especially in a politically charged educational 
context. ^ ^-"-^ 

He begins his argument with a brieiBTTt useful review of the 
histor> of those federally-funded educational programs that gave 
impetus to the modem development of Educational evaluation. He 
reminds us that Sen. Robert I^nnedy insisted on regarding eval- 
uation as a monitoring activity which would serve to increase 
political support for reforming the educational system so it would 
more adequately serve the poor. But at that time, the advocates 
of Planning, Programming, and Budgeting Systems (PPBS), Man- 
agement by Objectives ^MBO), and related analytical approaches 
to improving decision making, as represented by Robert McNamera 
in the Defense Department, were in the ascendancy. 

The advocacy .of ari^lytical accounting was attractive to many 
educational researchers who spearheaded the educational evalua- 
tion movement and who subsequently emphasized evaluation s 
analytical functions rather tlian the original monitoring role that 
Senator Kennedy intended. 

O'Shea then discusses some conditions that must be present or 
assumptions that must be met if evaluation is to fulfill its potential 
as an analytic tool for school districts. Briefly stated, educational 
methodologies are technically weak — we do npt know with suflR- 
cient clj(ty»the cause and effects in educational practice. What, 
for example, precisely makes for effective teaching and efficient 
learning? The precise linkage between cause and effect is still 
uncertain. Given this weak technology^ school districts function as 
institutional rather than as technic^ systems — they pay far closer 
attention to symbolic acts than to technical concerns. 
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Such an institutional setting, where the relationships between 
teaching and learning are hunches rather than immutable truths, 
is not an appropriate one in w hich evaluators should tr> to perform 
heroic, complex analyses. It follows that until more is knowTi about 
the causal lint^ge, evafluation Should realistically limit itself to 
achievement monitoring rather than complex analytical work. 

Drawing upon data from the CSE school district study, O'Shea 
shows that evaluation units in the sampled districts did indeed 
spend the vast majority of their time in monitoring rather thau in 
analytic activities. O'Shea suggests that if the units wish to become 
mvolved in instructional, i.e., technically oriented, evaluation they 
must heed Ebefs (1980) advice to adopt a methodology appropriate 
to that which they are evalu^ing, i.e., tc be more hetmeneutic 
than experimental, more qualitative than quantitative. They must 
also seek out measures which are sensitive to teachers* instruc- 
tional goals and which provide information useful for local instruc- 
tional improvement. 

Contribution and Importance 

Based on the preceding kinds of argument, O^Shea encourages 
districts wishing to inform instructional decision making to empha- 
size the use of instruments which are closely linked in content to 
the district's instructional programs. By u§ing instruments such as 
criterion-referenced tests school districts can get a more direct 
measure of the effectiveness of the instructional program than can 
be achieved with other kinds of tests. Finally, for those who v^sh 
to use evaluation for district-level decision making, O'Shea e^npha- 
sizes the selection of methods, which will usually be qualitative, 
which ^re responsive to the conditions and uncertainties of the real 
world of schooling, 
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School District Evaluation Units: 
. Problems and Possibilities^ 

v'David W. O'Shea 
• University of California, Los Angeles 

1, , ^ ' - 

• • . / ' INTRODUCTION 

'Widesprea.d^eniergence of school district evaluation units iji 
recent ye^'ars^l^ects ajocal response to mandatory^ reporting pro- 
pulsions of the* Elementary-: and Secondary' Education Act (ESEA), 
'^and of similsa- legislation in tbfe .individual states. ESEA requl&d ' 
participatin^states to secqre* from each school district, data on 
student acadJnic achievement by selected grades. The legislation 
also Tequired^ each district receiving federal funds 

, . to assess* the effect of its program and to identify weaknesses' 
' ' . as well as strengths of the project, thus serving as a tool for program 
. ' "^.revision andiipprovement (USCAN, p. 411, cited bfy David, 1978, 

Eighty-five percent o£* school district evaluation units; which 
now exist in 43 percent of the nation^s 750 districts enrolling 10,000 
pupils or more, have been established since 1965, the first year of 
ESEA (see Lyon, Doscher, McGranahan, & Williams, 1978). 
Curiously, however, while 'these evaluation units expend great 
effort on gathering achievement data, little attention is given to 
*the analysis of these data \<^ith a view to exploring the "weaknesses 
as well as strengths of the project," as legislators hoped. In prac- 
tice, district eval^ifetion units are more likely to engage an achieve- 
ment monitoring xdith^r than in analytic evaluation (see Kerins, 
1973, p. 76; Ebel, 1980, pp. 281-292).. Monitoring involves 
recording changes over time in average levels of academic achieve- 



»I am indebted to the following piprsons for their thoughtful comments on 
earlier drafts Adrianne Bank* Robert F. Boruch, Oscar Grusky, John W Meyer, 
Mary Ann Millsap, Richard Williams, and Lynne Zucker Also I am indebted to 
CSE for financial support. ;^ ' ' 
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mcnt, bv selected grade levels. Analytic evaluation attempts to ^ 
assess the contribution of specific instructional programs to student 
learning. 

The predominance of monitoring activity in the work ofschool 
district evaluation units provokes two questions, both of .which are 
' addressed in this chapter. First, what inhibits analvtic evaluations 

within school districts? Second, what might be done to facilitate 
more emphasis upon analytic evaluations? 

In exploring factors inhibiting analytic evaluations at the school 
district level we begin with a brief rev ievv of the origin of ESEA's 
evaluation requirements, and of the attempts to iniplement national 
evaluation of Title I programs, which was the largest component 

, ^ within ESEA, This review indicates that a m^^ jor inVi^KiHng f^<-^nr 

is disjiincture between the assumptions underlying most approaches 
to program evaluation and the principles underlying the practice 
of^mstructionxjl programs in the schools. EJ^idence of such dis- ' 
juncture is apparent in the content of th^ debates preceding 
implementation of ESEA in 1965, and also in the experiences of 
federally sponsored evaluations of instructional programs* which 
were funded under Title I of ESEA. To explore relevant evidence, 
first we turn to the original debate regarding ESEA evaluation. 

ASSUMPTIONS UNDERLYING 
ORIGIN OF EVALUATION REQUIREMENTS * 

Prior to passage of ESEA in l965,1nost evaluations within local 
school districts focused upon pupils, not upon instruction. The 
instructional program wa.s taken as given, leaving in question how 
well individual pupils were performing irt their effort to master the 
program content. Beginning in the 1950\s, the civil rights move- 
ment drew public attention to the fact that not onl^ did individual ^ 
pupils differ in their academic achievement, but whole groups * 
among racial minorities were performing at lower levels than the 
average for the majority white population. Specifically, by the 
1960's the phenomenon of the "achievement gap" was well docu- 
mented (.s^e especially Coleman, et aL, 1966). Pupils graduating 
from schools in low income, predominantly minority communities, 
were found to have average levels of achievement three grade.s. 
below the average fjr pupils from schools serving the more eco- 
nomically advantaged majority white population. 

Until the findings of Coleman, et al. (1966), emphasizing the 
dominance of socio-economic background upon pupil achieve- 
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ment, civil nghts leaders attributed the achievement gap to factors 
such ds the discriminator) allocation of financial resources among 
schools and/'or the absence-of -staff accountabilit> to parents. Con- 
gress responded with legislation addressing both possibilities. To 
improve resources, ESEA made available over three billion dollars 
annually for local schools. Title I of the Act, oriented to poverty 
area programs, received the largest proportion of the funds allo- 
cated. To facilitate staff accountability, the Act mandated state- 
level collection of pupil achievement data for each school. Finally, 
to keep Congress apprised of the impact of the money provided, 
ESEA called for annual national evaluations of the Title I program 
Requirements regarding collection of pupil achievement data 
were written into the Act in response to de^nands from Senator 
Robert Kennedy, then representing New York (see Bailey & 
MosHer, 1968, McLaughlin, 1975). Robert Kennedy supported 
ESEA on condition that the legislation include a reporting require- 
ment and "good faith administration efforts. to ht)jd educators 
responsive to their constituencies and to make educational 
achievement the touchstone of success in judging ESEA" 
(McLaughlin, 1975, p. 3). In Kennedy's opinion, if achievement 
data were available, parents would be in a position to monitor 
school performance and thus hold educators accountable for learning 
outcomes, creating an essentially political pressure for program- 
improvement. 

Kennedy's insistence upon the collection of achievement data 
was welcomed by officials in the Department of Health, Educa- 
tion, and Welfare (HEW), though nqt because they shared" his 
interest in facilitating parental pressure upon the schools. It so 
happened that coincident with the passage of ESEA, President 
Lyndon Johnson, fascinated by Secretary of Defense McNamara s 
tedmiques for controlling Pentagon budgets, c^nn-ouf^jped that the 
Defense Department's system of fiscal management, the Planning, 
Programming, and Budgeting System (PPBS), was to be installed 
throughout all sections of the executive branch (McLaughlin, 1975, 
p. 6). • 

^ To implement PPBS in HEW, the department which had overall 
responsibility for ESEA, William Gorham was brought in as Assis- 
tant Secretary for Program Evaluation (ASPE). From the outset 
the ASPE staff, unlike Senator Kennedy, took an analytic approach 
to evaluation. As Gorham (1967) wrote later (see McLaughlin, 
1975, p. 7): 



il3 



120 ' David W. O'Shea 



Title I of the Elementary and Secondary Education Act ... is 
essentially a vast experiment designed to find efifective ways of 
reaching disadvantaged children. We know that most school sys- 
tems have not been doing a very good job with these children, but 
there is no consensus among educators about how to do better. 

Obviously, Gorhdm and Kennedy had quite diflFerent approaches 
to program assessment in mind, ^ 

Kennedy thought of evaluation as a means to ensure that schools 
focused on the needs of the poor, to make sure that Title I 'worked.* 
Gorham s ASPE staff thought of evaluation as a way to find (\ut 'what 
works,* to identify the most effective compensatory strategies 
(^4^igWiiv4^75r^-p.^Z). ; 

Kennedy's approach involved achievement monitor^g, rather 
than program evaluation. Evaluation is analytic, involving what 
Thompson (1967, p. 86) calls "efficiency tests,*' focusing upon the^ 
worth, or effectiveness, of program processes in relation to pro- 
gram outcomes. 

Inhibiting the develop^pent of evaluation of instructional pro- 
grams as opposed to monitoring schooling outcomes are contradic- 
tions between the assumptions underlying analy tic evaluation and 
the nature of instructional processes in school settings. These con- 
tradictory assumptions become more apparent if one looks at past 
attempts at national evaluations of Title I programs. 

NATIONAL EVALUATIONS 

In enacting ESEA, Congress called' for both the development 
of school level achievement data, and an annual evaluation of the 
Title I program at the national level, withoyt specifying the form 
that such an evaluation might take. However, as noted earlier, the 
approach adopted for program evaluation was influenced by the 
government's desire to implement PPBS as the system for man- 
agement of all federal programs.^ Such an objective serves to focus 
evaluation upon cost-benefit analyses, orienting evaluation studies 
toward identification of the most cost-effective,' or efficient, means 
of improving student learning. To facilitate cost-benefit studies it 
is necessary to discover which program ^elements actually are effec- 
tive in promoting learning gains in classrooms. For this purpose 
the typical, though not exclusive, approach has-been to base eval- 
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uation upon' Experimental design, comparing the progress of stu- 
dents in funded programs with the progress of others in selected 
control groups. As Apsler (1977, p. 14) confirms, the experimental 
paradigm, so fruitful in the natural sciences, is '^advocated b> 
nearl> ever\ methodological discussion of evaluation research (see 
for example, Campbell, 1975, Houston, 1972, Suchman, 1967, 
Riecken & Boruch, 1975). 

In practice, this paradigm proved to be extremel> difficult to 
appl> to national evaluations of Title I programs, a factor inhibiting 

.downward ^^^ctention of analytic evaluation to the loCi^l district 
level. Nc!tionall>, an earl> attempt at anal>tip evaluation \^'as the 

^tud> oJ^mpensa^or>^^ucaiipj^ 

oTHEW b> the TEMPO division of General Electric. Data were 
obtained from schools in eleven districts considered to have 
"exemplar)" Title I programs. It was hoped that analysis of these 
data would determinejwhether "4'^*^^^*^^ amounts of money ^pent 
in different wajs would be significantly and differentially successful* 
in effecting academic achievement" (McLaughlin, 1975, p. 37). 

Unfortunately, the TEMPO study foundered. The main problem 
was that, as Ehel (1980, p. 290) concludes, in the case of schools 
it is almost^ innpossible to abstract a program from'its context and 
study it in isolation. Similarly, McLaughlin (1975, p. 37) reports: 

TEMPO analysts were able to identify neither a Title I population, 
nor Title I program, nor significant achievement gains that could 
be attributed to Title I funds. The incomplete, confusing, or non- 



^The objective at the federal level to rationalize program administration did 
not terminate with changes in the White House. Though the move to adopt PPBS 
begtin under Johnson, officials in OE were arguing for the same objective as 
recently as 1977, as evidenced Ij> the following comments of John \\\ Evans and 
Janict' K Anderson (Evans & .\nderson, 1977, p 160), who were senior officials 
in the Office of Planning, Budgeting, and Evaluation at the U S, Office of 
Education 

Since the late sixties. Congress has become increasingly disillusioned with the policy 
, relevance wf iriDSt general or basic research prtxluced through the open-ended grant 
process,, and it h.»;i>leirianded that federal agencies produce more immediately useful 
researcbevaliiafjon results Mission -based federal agencies have responded by moving 
rrujre in the applied direction, and in the* process have ass(!Vnl>led technically trained 
evaluation staffs to design and closely monitor the conduct of evaluations and other anal 
yscs. They are part of a larger trend, wilb other elements being the emphasis on 
aLCouittability . management by i^bjectives, program planning and budgeting, and the 
crealion-of the new budget Lommittee s procedures and ceilings within the Congress 
The basic thrust of this larger treiul is a belate<l effort in ifie social program area to 
rationalize fe<leral policies, and the allocation nf scarce resources 
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existent records at the local level made it difficult for TEMPO 
analysts to determine which services were being purchased with * 
Title 1 dollars, or to specify the group receiving special attention 
through Title I. In practice, there seemed to be no ceal Title I pro- 
gram to evaluate.^ 

BasicaH), confronting the staff of TEMPO, and of subsequent 
Title I evaluations, was the fact that the logic of evaluation using 
the experimental paradigm is at odds with the logic underl>ing 
schools."* This same conflict of logics also frustrates attempts at the 
school district level to develop formal program evaluation. 



A central requirement of an i/npact, cost-be^iefit study— the ability to tie inputs to out- 
puts—docs not coincide with the operational reality of Title I An attempt to trace the 
flow of Title I dollari to specific programs and outcomes is beset with problems In what 
.\lichael Kirst calls the "Byzantine world of school accounting," it is difficult if not 
impossible to trace the course of Title I dollars through the school system Some cities, 
especially larger cities, have over 100 sources of income As the number of revenue 
sources increases, the ability of^evaluators to identify the impact of any single source 



^In discussing McLaughhn's (1975) study of ESEA, House (1978) draws con- 
cjusions similar to those offered m this chapter. Commenting upon the approach 
to Title I evaluation, House (1978, p. 388) concludes that. 

The evaluation policy developed by the federal government reflected a particular ide- 
ology It reflected the belief of systems analysts and economists that evaluation should 
b^ use(l primarily to detect the most efficient programs This concern for efficiency led 
to evaluations that could find no differences between the new programs and those 
already existing in the public schools ^ 
The result has been that educational funds have l>een constrained at the federal level 
for lack of visible results and education has been discredited for its inability Actually 
the lack of results reflects more the type of evaluation employed than the quality of the 
educational programs. 

The ultimate danger of an approach to evaluation embodying a logic at odds with 
the reality of .vchooling is, as House (1978, p. 392) warns, that whde evaluation 
IS initiated in order to insure the success of educational programs, it can generate 
pressures directed toward the imposition on the schools of programs designed to 
ensure the success of evaluations. In brief, evaluation that assumes schools con- 
form to technical rationahty is likely to have the effect of absorbmg the normative 
systems of schotjls mto the system implicit in the experimental [faradigm under- 
lying most evaluations.^ 
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CONFLIQriNG LOGICS 



Insight.into the basis for these conflicting logics is provided b> 
Apsler's (1977, p. 15) comnnents; 

Although the needs of action programs might be ver> specific, such 
OS to raise children's test scores or to reduce the incidence of crime, 
the procedures b> which theii^ ends will be attained are generally 
quite unspecific. Unibrtunatel>, little, is known )et about hqw social 
action programs can raise children's test scores, reduce crime, and 
, so on. As a result, social action programs often become operational 
w ithout clear approaches or means of attaining their goals. Not sur- 
prisingly, the actual operations of a social action .program ma> then 
be primarily determined by intuitions and exigencies. The con- 
fusing situation that develops — in wJiich the approaches are poorl> 
thought through and in which approaches fr-equentl> var) in response 
to recurrent crises and changing intuitions — presents enormous 
difficulties to the evaluators. 

Such confusion is especiall> prominent, o1ie might add, when 
one operates within the constraints of the experimental, or evea 
the quasi-experimental, paradigm.^ Evaluation according to the 
experimental paradigm, with its assumptions about treatments 
causally related to outcomes, follows the logic of technical ratio- 
nality. Schools cannot fit this mode, as educators lack a theory on 
the basis of which they can specify which instructional interven- 
tions are likely to produce -specific learning outcomes. Were sucH 
a tHeory available it would guide specification of the interventions 
necessary to reduce the substantial gap in average levels of aca- 
demic achievement between schools serving low and middle- 
income communities. As Averch et al. (1962) determined, educa- 
tional research has not yet identified "what works." Instruction, 
like parenting, remains more an intuitive and practical art than 
science, as I^ortie (1975) argues. Like parents, teachers follow the 
normatively grounded logic of practical rationality. 

However, as Bernstein (1975, p. 64) reminds us, in our tech- 
nological society we are dominated by the values of a culture that 
IS grounded in theoretical knowledge, and committed to technical 
rationality. In this context, it is not surprising that most evaluators, 
and policy makers, subscribe to the notion that all institutions 
should operate in <i technically rational mode, and be ev aluated in 
terms 'of the experimental paradigm. However, as Goodlad (1975) 
argues, and ako House (1978), once you begin thinking about'* 
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schools in the framework of the experimental paradigm, with its 
dependent and independent variables, there is a danger of 
absorbing the reality of schooling into the model of a technically 
rational system epitomized b> the modern factory.^ Factories 
embody technology based .upon tested knowledge of means-ends 
' relationships. In designing an automobile plant, for example, plan- 
ners draw upon ai)od> of knowledge that is sufficiently developed 
to serve as a basis for specifying in advance all the functions that 
must be performed in order to transform an array of raw materials 
mto a finished automobile. Systems embodying technical ratio- 
nality are greatly extending man s donaination over nature, facili- 
tating the exploration of space, and providing the ^terial basis for^ 
contemporary industrial civilization. • ^. 

Schools, however, are not of the same organizational genre as 
factories and other technically rational systems. Given the enor- 
mous prestige of the latter systems there is, of course, continuous 
pressure for schools to move toward the technically rational mode. 
Inhibiting such movement in the field of schooling is the absence 
^ of the ingredient most necessary to technically rational systems, 
tested knowledge of the relationships between instructional means 
and educational ends (see Averch et al, 1972). In effect, as Dreeben 
(1970) reminds us, we lack a technology of instruction, a condition 
leading Metz (1978, p. 20) to conclude: 

Technology, then, is a major problem for the public schools. They 
^are faced with the task of creating changes in diverse raw material 
through processes that are poorly understood, in the absence of any 
universally effective means, and without any trustworthy way of 
measirt-mg the succQSis or failure of whateV,er methods |[iey finally 
apply. 

Schools^ weak technology, coupled with their highly variable 
material, in the form of pupils, constrains a decentralized type of 
organizational structure. ^Coordination is achieved not by staff 
implementing a specific technology l)ut by the collective enact- 
ment of a moral order, or institutionalized social system (see Metz, 
1978, p. 30). As Meyer (1977)^ argues, schools are institutional 
rather than technical organiziiUons. Coordination of activities is— 

*One then becomes entrapped in the phenomenon Wise (1977, p 44) lcil)els 
hyperrationality.** In essence, hyperrationahty represents an assumption that the 
logic of technical rationahty is at work when, m fact, it is not 
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achieved primarilv b> basing school programs upon personnel 
who, having been trained, screened, and certified b> state approved 
teacher-education programs, share a common understanding of the 
professional duties to be undertaken, and of the rules governing 
performance of these duties. Schools, as imtittitional organiza- 
tions, articulate a particular set of rules, many of whi^ch are 
embodied in the educational codes of the different states, which 
are the political entities cQnstitutionallv empowered to authorita- 
tivel> express societal expectations regarding the wa>s in which 
emerging generations ought to be inducted into adult society. 

Ev aluation of the o rganizat ions w ithin which the induction process 

IS institutiondized, and of the specific programs witFiin these 
organizations, calls for a different approach than that which is 
appropriate for technical organizations. 

TECHNICAL VERSUS INSTITUTIONAL ORGANIZATIONS 

The major determinant of organizational type, in Me>er's (1977) 
opinion,i.is the nature of the social environment within which the 
organization evolves. Technical organiziitions, such as factories, 
evolve in environments with complex technologies. B> contrast, 
institutional organizations, such as schools, evolv e in environments 
with ver> limited technologies hut with elaborate institutional 
rules. The ta,sk of a technical organization is to coordinate and con- 
trol technical w^ork. The task of an institutional organization is to 
create strtictures that conform with institutionalized rules gener- 
ated over time within societ> to order the maintenance of the social 
system. 

Technical organizations, in their pure form^ are predominantly 
cybernetic systems, oriented to goal attainment, and regulafed b> 
•feedback from their own results (see Habermas, 1970, pp. 87-93). 
By contrast, institutional organizations are socialization systems, 
oriented to getting participants to internalize the system's consti- 
tutive norms, or institutionalized rules. 

In evaluating technical organiziitions, priority is given to as.sessing 
the cost-effectiveness of alternative means to achieve specific out- 
comes. When institutional organizations are being evaluated prior- 
ities shift. The dominant concern becomes the degree to which 
organiz<itioii structure and proce.sses conform .to the values to 
which the organiAition is expected to give expression. The effec- ^ 
tiveness of a school in promoting the academic achie;Vement of the 
pupils plays a minor role in .school accreditation. The priorities are 
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facilities, stafif qudlificatioji, teacher-pupil ratios, and the scope and 
content of the educational program. ' ' * 

Not onl> do technical and institutional organizations differ in 
terms of the premises upon v^^>ich"the> are structured,, but their 
structural Characteristics take on radicall> different forms. Knowl- 
edge of means-ends relationships allows the administrators of tech- 
nical organizations to direct all operations from the top down. By 
contrast, in the case of iftstitutional organizations, the relative 
absence of technical knowledge constrains administrators to leave 
operations to the discretion of staff at the operational, or technical, 
ie:vLeL^Oiie-coiisequence-4n the-ca^e^^f schools, as •Metz-(197&, - 
p. 21) concludes, is that: 

.The persons who perform the actual work of the organization need 
' to be given relatively large and diffuse tasks with the right to make 
important decisions independently as they use their intuition to 
adjust their methods to the requirements of each specific instance 

While technical organizations are highly centralized, with the 
administrative and technical, or production, levels tightly coupled, 
schools, as institutional organizations, are decentralized, with 
administration and the technical or teaching level lopsely coupled,^ 
at least with regard to the formal work o|^the schools, the instruc- 
tion of students (see Weick, 1976; Meyer, 1977.)^ School princi- 
pals, for example, though including instructional leadership among 
their responsibilities, seldom visit classrooms or involve theni- 
selves in the details of instructional processes. The latter are del- 
egated almost entirely to teachers. 

In summary, therefore, schools, as institutional organizations 
combine a relative absence of technical rationality with a^ieofre- 
spondmg emphasis upon practical rationality, and loosely coupled 
relationships between administrative and instructional levels, three 
conditions that constrain school evaluations toward program . 
monitoring. 



* Others concerned with the nature of schools as organizations also have noted 
their loosely coupled structure. See, for example, Bidwell (1965), Dreeben (1970), 
Metz (1978), Wolcott (1977). 
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PROGRAM MONITORING 



Given that institutional and technical organizations operate on 
the basis of different premises, approaches to evaluation that are 
appropriate for the one t>pe pj'ove inappropriate for the other. In 
exploring the theory of organizations Thompson (1967, pp. 83-101) 
points out that the appropriate mode of organizational assessment 
is determined b> two variable factors. (1) the degree to which there 
IS clarit> regarding the effects, or outcomes, desired, (2) the degree 
of knowledge available regarding the means that actually can pro- 
duce the desired outcome. 

The same principles apply to program evaluation. *Where it is 
clear just .w hat outcome is desired, and there is full knowledge of * 
the means needed to attain this outcome, administrators can both 
prescribe and control the relevant means, In this context, evalu- 
ation can focus upon the cost-ejfedtiveness, or efficiency, of the 
program. 

Where knowledge of means/ends, or cause/effect, relations are ^ 
incomplete, as with schools and their programs^ 

.... the efTiciency test is inappropriate, for there is no way of 
assessing the net effect of causal action. In this case, the appropriate 
test is not the economic one, but thef instrumental one — whether 
a desired state of affairs is achieved (Thompson, 1967, p. 86). • 

In the case of schools, knowledge of the causes of student aca- 
demic achievement are notably incomplete, a condition that leads 
Ebel (1980, p. 288) to conclude: 

In seeking to evaluate an educational program, we ma> be led astray 
if we try to follow too closely the model of the scientific research 
study. An educatioTial program is not a stable, natural phenomenon 
with built-in operating characteristics that people may discover and 
put to use, but not alter in any fundamental way. An educational 
program is a huTnnan artifact, highly complex, infinitely variable, and 
subject to incessant change. It defies precise definition or accurate 
measurem^jnt. The outcome of a precisely controlled scientific study 
of an educational program is almost certain to be either inconclusive 
or misleading. 

Logically, therefore, from Thompson s (1967) perspective, eval- 
uation of educational programs should be conducted in the instru- 
mental mode. In effect, this mode corresponds to achievement 
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monitoring. Rathe/than struggle with tlfe^ethodological hazards 
associated with attempts to identify "what works," in principle, as 
Ebel (1980, pp. 281-292) proposes, the focus shifts to whether a 
program works. In practipe, this monitoring approach already is 
the one most cofnmonl> adopted by school district evaluation 
units. For example, among the 263 districts responding to the CSE 
survey, 75 percent ranked as their most important task the gath-^ 
ering of data on pupil achievement. Task importance in this instance^ 
was measured b> the proportion of the evaluation unit's time given 
to selected data-collection activities, as shown in Table 1. 



Table L Percent of School District Evaluation Units, Ranking 
Select ed Activities in 'Order of Time Taken in 1977-78 (N=263) 

Rank Order of Time 



Activities 




Taken > 




^ Gathering data "ipn: 


1 


2 


3 


Student achievement \ 


75% 


16% 


4% 


Relationship between school/classroom 


2% 


15% 


8% 


characteristics and student 








achievement 




>\ 




Language dominancef^f studetots , 


2% . 


V2% 


11% 


Relationship between student socio- 


1% * 


8% 


8% 


economic status and achievement 








Student socio-economic characteristics 


0% 


2% 


8% 


Handicapped students 


1% 


11% 


6% 


Relationship between students' race/ 


0% 


3% 


13% 


ethnic background and " 








achievement program 








Other 


.19% 


33% 


42% 



That these data are used for monitoring achievement outcomes 
rather than for analytic evaluation of the instructional program is 
indicated by the very low proportion of districts assigning either 
first, second, or third priority to securing data on the relationship 
between school/classroom characteristics and student achieve- 
ment. Further, any systematic evaluation of the effects of instruc- 
tion upon student achievement must take account of student socio- 
economic status, the major predictor of academic performance (see 
Coleman, et ai, 1965). Despite the necessity to control for socio- 
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economic status in any meaningful analysis, few districts allocated 
it either a first, second, or third level of importance. 

Confirming the^findings of the CSE survey are results frDm 
David s (1978) study of local uses oi Title I evaluations in 30 
**exemplar>" school districts across six states. David found few 
district evaluation units viewing the collection of standardized 
achievement test data as relevant to the ev aluation of instructional 
programs in the schools. Rather, such data were seen as serving 
informational, or monitoring, needs of the state and federal levels 
of educational governance. Locall>, the rationale for continued use 
of standardized tests was that the> were the simplest means of 
responding to state and federal reporting requirements (David, 
1978, p. v). 

• EVALUATING INSTRUCTION 

Given that current activities of school district evaluation units 
generall> cover the need to monitor program outcomes, providing 
annual data on pupil achievement relative to national norms, the 
problem remains of actually evaluating the adequacy of the instruc- 
tional procedures underlying pupil learning gains. Regarding this 
problem of instructional evaluation David (1978) asked respon- 
dents in the 30 districts she studied how they would demonstrate 
the success of their pi'ograms. Preferred approaches, for botlj staff 
and parents, emphasized utilization of criterion-referenced, or 
"curriculum-embedded and other skill tests" (David, 1978, p. vi). 
Such tests incorporate items projected forward from the actual cur- 
riculum content. Test results thus bear directly upon the extent 
to which local instructional objectives are being attained, allowing , 
teachers, the ultimate instructional decision-ijnakers, to determine 
which curriculuni components need more, or less, emphasis. 

Devolution of responsibility for testing to the teachers at the 

instructional level of the school certainly respects the logic of 

schools as institutional organizations, with teachers conducting 

instructional activities relatively autonomously of the administra- 

* tion. To make this alternative succeed, Ebel argues, 

I 

local school personnel need to receive training in evaluation pro- 
cedures to the point where they can do the job as adequately as an 
external evaluator can be expected to do it. Practical evaluation 
I techniques are not so highly technical or so remote from typical 
school operations as to require a separate profession of evaluation. 
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People competent to design programs and to teach should also be 
competent to assess the results of teaching (Ebel, 1980, p. 284). 

Ebel's approach addresses the problenn of helping teachers to 
improve instructional programs on the basis of their own 
evaluations. 

If one wishes to go beyond achievement monitoring, and also 
pursue program evaluation from a perspective broader than that^ 
of the individual classroom teacher, it is necessar> to adopt a meth- 
odology appropriate to the loosel> coupled structure of schools, 
drawing upon the hermeneutic rather than the experimental tra- 
dition of research design. Essentially, this involves a methodology 
emphasizing qualitative rather than quantitative studies of instruc- 
tional programs. 

^ QUALITATIVE EVALUATION 

The alternative to what Patton (1978, p. 204) calls "the dominant 
hypothetic-deductive paradigm" is being applied to evaluation by 
persons such as Parlett and Hamilton (1976), Robert Stake (1975), 
and Kenneth Strike (1972). While "the natural science paradigm 
aims at prediction of social phenomena, the holistic-inductive, 
anthropological paradigm aims at understanding of social phe- 
nomena" (Patton^, 1978, p. 204). This paradigm draws upon the 
more qualitative methods and perspectives of- phenomenology, 
symbolic interactionism, ethnomethodology , and . anthropology 
rather than the quantitative. 

Specific objectives associated with qualitative evaluation are 
outlined by Parlett and Hamilton (1976, p. 144) as cited by Patton 
(1978, p. 209). These objectives, in relation to ^ school program, 
are likely to include: 

. . . how it operates, how it is influenced by the various school sit- 
uations in which it is applied, what those directly conc^me.d regard 
as its advantages and disadvantages, and how stude*nts* intellectual 
tasks and academic experiences are most affected. It aims to dis- 
cover and document what it is like to be participating in the scheme, 
whether as teacher or pupil, and, in addition, to discern and discuss 
the innovations* most significant features, recurring concomitants, 
and'critical processes. In short, it seeks to address and to illuminate 
• a complex array of questions. 
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In practice, implementation of such **illuminative" evaluation 
requires specialized personnel, thus restricting its use to evalua- 
tions sponsored b> state or federal agencies, rather than local dis- 
tricts, though the latter ma> well opt to encourage universities in 
their areas to expldre this qualitative approach. 

CONCLUSION 

At the outset of the chapter two questions were raised. (1) What 
inhibits anal>tic evaluations of instructional programs at the school 
district level? (2) What might be done to facilitate more emphasis 
upon analytic program evaluation? 

The major inhibiting factors appear tc be: 

(1) c*ooflict betw een the promise and the performance of national 
evaluations, typically based upon the experimental research 
paradigm; 

(2) attempts to use the experimental research paradigm for eval- 
uating instructional programs in schools, despite the fact that 
schools are not teohnicilly rational but institutional 
organizations; 

(3) utilization of standardized norm -referenced tests for reporting 
the outcomes of externally funded programs, despite the low 
credibility ^ of these tests as a means of local program 
assessment. 

School district evaluation units tend to remove themselves from 
the cross-pressures inherent in the above contradictions by opting 
out of any sustained commitment to program etaluation, focusinb 
instead upon monitoring program outcomes in the form of st'iklgnl 
achievement, administering tests to participants in funded ^ro-^* 
grams, and to pupils at selected grade levels across alV schools. Test 
results are reported to state^ educati9n departments, to other 
funding agencies, and to the public locally. 

While monitoring has utility in locating schooling outcomes of 
one district relative to another, and in identifying temporal trends, 
it says nothing about existing instructional procedures. To evaluate 
relationships between instruction and learning outcomes requires 
the adoption of evaluation methodologies not in contradiction with 
the" nature of schools as institutional organizations. Such metKod* 
ologies are available, as discussed here, for use by both the indi* 
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vidual teacher, and b> external evaluators, but require commitment 
- to criterion-refereYiced tests for teacher-based evaluation, and to 
qualitative methodologies for external evaluation. 

The danger to avoid is that identified b> House (1978), and dis- 
cussed in more general terms b> Habermas (1971). Thi3 danger is 
that rather than adopt evaluation approaches appropriate to the 
nature of schools and school processes, pressure will be exerted 
m the direction of making schools into technicall> rational orga- 
nizations, m the belief that previous null findings reflect the inad- 
equacies of schools, rather than the inappropriateness of current 
methods of evaluation. 
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The chapters in this monograph have incorporated four author's 
interpretations of, or reflections on, the CSE survey data of school 
district evaluation units, specifically as the data relate to evalua- 
tion s role in ichool district operations and decision malang. Because 
the authors are sociologists they examine school districts and their 
evaluation units as social entities, as one would expect, common 
themes run through their papers. The authors^ diflFer mainly in the 
emphasis they give to these common themes. 

In this concluding section, we indicate three o£ the common 
themes which form the core of an argument that should be con- 
sidered by school district evaluators and by evaluation researchers, 
especially those who are interested in increasing the power of eval- 
uation in school districts. We will also discuss these themqs in 
relation to some of the current work at the Center for the Study 
of Evaluation, and speculate on their implications for further 
research and development in the field of evaluation. 

Theme 1: School districts are institutional organizations. 



School districts are, with regard to th^rfinstructional activities, 
essentially institutiqnal rather than technical organizations. An 
important reason for this is that schooling has a weak technical 
core — little is known with certainty about how teacher behavior 
and educational programs afiect pupil learning. Thus, it is difficult 
to link the technical core — a fundamental feature of technical 
organizations — to the organizational structure. As a consequence 
of, the institutional nature of schools, it is difficult for evaluation 
findings to be seen as a strong tool for school or educational pro- 
gram improvement. It is difficult to make credible the precise anal- 
ysis of phenomena that are themselves not seen as precise. 

Also, there are few incentives for school districts to become 
more technic^ly oriented thereby enabling them to use evaluation 
data for instructional decision making. School districts have been 
largely successful as institutional oi;ganizations, the public has been 
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willing to accept symbolic representations, e.g., tenured teachers 
with advanced degrees, as indices that the schools have quality. 
Much of the technical work performed b> school personnel — that 
is, work related to instruction and teaching — is kept shielded from 
public view. 

Theme 2: Evaluation units have a weak position within school 
districts. 

I 

Evaluation units often have little power within school districts. 
Most such units originated in response to accountability- demands 
from external sources, and their work in testing and evaluation 
tends to emphasize external "signaling." The information desired 
b> external funding agencies, because it is designed to ensure com- 
pliance or to make comparisons, is different from that which could 
serve internal instructional needs. Since most evaluation units 
have onl> the time or staff to provide one kind of information, they 
primarily serve governmental or community clients on behalf of 
their districts, and they do only that monitoring and testing which 
appear to be required. Many evaluation units thus face a dilemma, 
because they do not producQ^^much information that., is valued 
internally, they occupy a peripheral status vis-a-vis other district 
operations. Because of this peripheral status,, they are not in a 
position to influence district policies or operations. Nor are they 
given much opportunity to demonstrate what they might do in^ 
relation to the improvement of management or instruction. 

Theme 3: It is diiBcuIt to produce evaluative data of utiUty to within- 
district (fecision makers. 

The low organizational status that evaluation units have may be 
exacerbated by the professional stance of some evaluators. Eval- 
uators may be seen as — and may see themselves as — having an 
"objective, analytical and rational" orientation. This orientation 
can bring them into conflict with district and school administrators 
who live in a world of clashing political and economic forces, and 
with teachers who must work in the constraints of the classroom. 
District administrators must often walk a tighti'ope between con- 
flicting constituencies, in given situations, they may not or will not 
use the findings from data-based analyses of programs they are 
considering or operating. Put another way, school administrators 
may view the data from evaluation or from testing programs as only 
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one factor to consider in making decisions. They operate with an 
institutional/procedural/situation-specific ratibnalit> as distinct from 
a t^chnical'method-based/generalizable rationality. Teachers may 
jesent time taken from instruction by an evaluation they see as 
having litde relevance to their work. 

Thus, the political as well as the organizational context of a 
school district places 'severe* limits on the internal utility of the 
work of evaluation units. The credibility of evaluators is further 
diminished because of the indeterminacy of cause and effect in 
education, as previously mentioned. Because the "technical core — 
that is, certain knowledge about the causal nature of the relation- 
ship between teaching and learning — is weak, instructional rec- 
ommendations based on evaluation or test findings are at best 
tentative. Conclusions from evaluative data can be discredited. v^th 
relative ease by those who, for political, financial, economic, or 
professional reasons, are in disagreement with the findings. 

Implications for the Future 

These three themes would seem to explain why evaluative data 
are not now used internally, uithin the central office, schools or 
classrooms. Evaluation units, as organizational entities, face chal- 
lenges with regard to integrating their "signaling" function v^th 
the production of data-based information related to the technical 
work of schooling. ' 

Moreover, new policies and shifting conditions may prove to be 
an additional challenge for district evaluation units that seek a 
more stable and valued place in school district decision making. 
The current move awa> from external categorical aid programs 
towards block grant fmiding gives local educational ageruries greater 
discretion oyer how and where to target external funds. The 
freeing up oif these funds may also relax some of the specific eval- 
uation requirements that have typically accompanied categorical 
grants. Given such freedom of choice, will school district admin- 
istrators choose to invest their funds in supporting the work of eval- 
uation units and redirecting the evaluation towards internal 
problems? In the field work CSE has recently conducted in school 
districts with evaluation units we have not yet found strong, broad- 
based support for evaluation, although there are a number of dis- 
tricts in which specific administrators and school board members 
ask for and use the^evaluation units' work. Such advocates might 
look favorably on funding evaluations from local or internal funds. 
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We doubt, however, that these decisions can prevail for long when 
the> are placed in competition against various larger political and 
special interest groups, e.g., unions, parents of handicapped chil- 
dren, who want a share of an increasingly shrinking school district 
budget. It ma> alsote difficult for school district evaluation units 
to restructure their own thinking so as to produce the kind of 
reports that will capture the attention of decision makers and 
strengthen the resolve of administrators and Board members to 
retain th&ir services. While the academic communit> continues to 
produce better data collection instruments and evaluation designs 
and to upgrade the qualit> of evaluators* training, these eflPorts by 
themselves are not likely to solve the problem of evaluation uti- 
lization Within school district offices. 

Thus, declining resources and considerable external political 
turmoil contribute to the school administrator s problems in trying 
to satisfy simultaneous demands for reform and for retrenchment. 
Powerful individual and group interests are at stake and educa- 
tional administrators and policy makers may be forced towards 
political accommodations rather than empirical data when reaching 
decisions. Administrators and policy makers may not yet have 
come to appreciate the contribution evaluation can make to their 
work, they may feel that evaluations ^re risky, that they have 
potential for upsetting organizational equilibrium and so make the 
decision makers' tasks more difficult. 

' In spite of these challenges, we believe that the present situa- 
tion in school districts offers new opportunities for enhancing the 
relevance of evaluation and testing for local decision-making pur- 
poses properly seized by thosp in district leadership positions. 

Communities are indeed plagued by declining sources of rev- 
enue. They therefore are pressing school administrators to dem- 
onstrate that public schools can teach children to acquire basic' 
skills. Parents, concerned about academic and vocational oppor- 
tunities for their children, often interpret their own child's per- 
formance on standardized tests as indicators of the schools' ability 
to teach. Simultaneously, district gidrfiinistrators themselves are 
M becoming ev fen more aware of their need to monitor both the long- 
term patterns of students' test scores as well as to analyze any given 
year's test or evaluation data in a way that suggests policy, admin- 
istrative, or instructional remedies. 

At the same time that school districts experience this increased 
press for description and diagnosis of student learning, many 
•administrators do find that they do possess, within the district, the 
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capdcit> for responding. The past fifteen years of federal mandates 
for evaluation have helped equip some, if not all, districts with 
competent personnel who have data collection and data analysis 
skills which can be re-directed for internal use. If federal require- 
ments for evaluation slacken, evaluation units ma> have the oppor- 
tunity to justify their^ existence by responding to 'internal 
management needs. 

Our work has led us to conclude that evaluation units have great 
potential for providing timely, technically sound, and useful ser- 
vices to school districts. They, and those in district leadership 
positions who beheve in the value of systematically collected data, 
might consider some promising, orientations derived from CSE's 
current fieldwork. 

1. Multiple definitions of evaluation units roles. Evaluation 
units can play many roles in school district functioning even 
though many in the past seemed limited to administering 
tests and monitoring compliance. Evaluation units and deci- 
sion makers should explore common interests and ways in 
which evaluation, through new and different techniques, can 

. serve the multiple n^eds of a variety of qonstituencies. For 
example, a common observation in school districts has been 
that evaluation units only serve those programs with external 
evaluation funds. Many teachers in those programs feel that 
such evaluations are inconsequential or even detrimental to 
their work. Evaluation anits might well begin exploring ways 
. in whjch they can serve immediate teacher needs thereby 
solving rather than exacerbating teacher problems. In this 
process, the support of central office leadership is essential. 
"Idea champions" who have the vision to see how evaluation 
findings may serve to improve teaching, and learning at the 
classroom level can work out situation-unique solutions to 
the management problems of linking evaluation to interna! 
decision making. , 

2. Imjrroved comrnunicdtions. Part of evaluatidn's^previouS lack" 
of a district constituency may have resulted from the failure 
of evaluators and decision makers to understand one another's 
world and to recognize the strengths and weaknesses of eval- 
uation as a decision-making tool. Supportive^chool admin- 
istrators and evaluators must now take new steps to assess 
the role evaluation has played and can play in district activ- 
ities. Such an examination should include efforts to under- 
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stand the contributions and complexifies of linking data with 
instructional change. Evaluators must become sensitive to 
the need to follow up their data collection activ ities w ith data 
dissemination activities. They must also participate in the 
organization problem-solving process whicii should follow 
evaluation. It. seems very unlikely that school districts will 
spend diminfshing funds on any services that;^are viewed as 
peripheral by those who control the funding decisions. Eval- 
uators should become central and influential in the internal 
management of the district and in the improvement of 
instruction. 

3. Evaluation and instruction. While the monograph authors 
have argued that education still has a weak technical base, 
Ihe work of several researchers and developers (Fisher et ai, 
1978; Stallings, 1980; Bloom, 1980; Hunter, 1971;) can, in 
our opinion, be seen as the foundation of an emergent tech- 
nical core. If this is the case, and some school districts are 
beginning to identify it as such, then evaluation activities can 
indeed begin to contribute to understanding the relation- 
ships between processes and outcomes within given settings. 
School district ey^iluation units can work with the district's 
ihfitructional program and contribute to its improvement. , . 

We believe that the evaluation community outside of and within 
school districts is likely to become, in the next ten years, more 
intimately involved than in the past with the definition, clarifica- 
tion, and resolution of important educational problems. The iso- 
lation- of evaluators is ending. Individuals interested in evaluation 
and testing are being forced to grapple with a broad range of 
serioi^s concerns in education today. They are being asked to make 
a contribution to the knowledge base underlying learning, instruc- 
tion, curriculum, and management. Evaluation and testing per- 
sonnel, as they function in school districts, are likely to become 
more pluralistic and interdisciplinary. These individuals will come 
to po.Susess a broad range of knowledge and talents. They will come 
to understand with sensitivity and empathy the context within 
which other educators work. T\my will come to anticipate the 
impact of their activities on the people who are evaluated and on 
those who use evaluation findings. We believe that they have the 
potential for making a difference. 
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